Assume that $a\lt0$ (otherwise, the integral diverges).
Real Variables
Letting $c=\frac{b}{\sqrt{-a}}$ and $u=\sqrt{-a}\,x$, we get
$$
\int_0^\infty e^{ax^2}\cos(bx)\,\mathrm{d}x
=\frac1{\sqrt{-a}}\int_0^\infty e^{-u^2}\cos\left(cu\right)\,\mathrm{d}u\tag{1}
$$
Now take the derivative with respect to $c$ (which is fine since the integral converges absolutely):
$$
\begin{align}
\frac{\mathrm{d}}{\mathrm{d}c}\int_0^\infty e^{-u^2}\cos\left(cu\right)\,\mathrm{d}u
&=-\int_0^\infty e^{-u^2}u\sin\left(cu\right)\,\mathrm{d}u\\
&=\frac12\int_0^\infty\sin\left(cu\right)\,\mathrm{d}e^{-u^2}\\
&=\left.\frac12\sin\left(cu\right)e^{-u^2}\right]_0^\infty-\frac c2\int_0^\infty e^{-u^2}\cos(cu)\,\mathrm{d}u\\
&=-\frac c2\int_0^\infty e^{-u^2}\cos(cu)\,\mathrm{d}u\tag{2}
\end{align}
$$
That is, $I'(c)=-\frac c2I(c)$. Therefore, $I(c)=\frac{\sqrt\pi}2e^{-\frac{c^2}4}$. Thus,
$$
\int_0^\infty e^{ax^2}\cos(bx)\,\mathrm{d}x
=\frac{\sqrt\pi}{2\sqrt{-a}}e^{\frac{b^2}{4a}}\tag{3}
$$
Contour Integration
$$
\begin{align}
\int_0^\infty e^{ax^2}\cos(bx)\,\mathrm{d}x
&=\frac12\int_{-\infty}^\infty e^{ax^2}\cos(bx)\,\mathrm{d}x\\
&=\frac12\mathrm{Re}\left(e^{\frac{b^2}{4a}}\int_{-\infty}^\infty e^{a\left(x+i\frac{b}{2a}\right)^2}\,\mathrm{d}x\right)\tag{4}
\end{align}
$$
Use the contour
$$
\gamma+R=[-R,R]\cup\color{#C00000}{[R,R+i\frac{b}{2a}]}\cup[R+i\frac{b}{2a},-R+i\frac{b}{2a}]\cup\color{#C00000}{[-R+i\frac{b}{2a},-R]}
$$
and Cauchy's Integral Theorem to compute
$$
\int_{\gamma_R}e^{az^2}\,\mathrm{d}z=0\tag{5}
$$
The integral over the vertical segments (in red) vanishes as $R\to\infty$ and the integral over the horizontal segments (in black) tends to
$$
\int_{-\infty}^\infty e^{ax^2}\,\mathrm{d}x-\int_{-\infty}^\infty e^{a\left(x+i\frac{b}{2a}\right)^2}\,\mathrm{d}x\tag{6}
$$
Therefore, $(5)$ and $(6)$ imply
$$
\begin{align}
\int_{-\infty}^\infty e^{a\left(x+i\frac{b}{2a}\right)^2}\,\mathrm{d}x
&=\int_{-\infty}^\infty e^{ax^2}\,\mathrm{d}x\\
&=\frac{\sqrt\pi}{\sqrt{-a}}\tag{7}
\end{align}
$$
Combining $(4)$ and $(7)$ yields
$$
\int_0^\infty e^{ax^2}\cos(bx)\,\mathrm{d}x
=\frac{\sqrt\pi}{2\sqrt{-a}}e^{\frac{b^2}{4a}}\tag{8}
$$