Albeit tedious and possibly unnecessary, you can certainly evaluate $\displaystyle\int_{0}^{\infty}\exp\left(-x^{2}\right)\cos\left(ax\right)dx$ using your rectangular contour on the first quadrant of the complex plane. For $a \in \mathbb{R}$, we will prove using your methods that
$$\int_{0}^{\infty}e^{-x^{2}}\cos\left(ax\right)dx = \frac{\sqrt{\pi}}{2}\exp\left(-\frac{a^{2}}{4}\right).$$
Let the given integral in question be $I$. Then
$$
\eqalign{
I &= \int_{0}^{\infty}e^{-x^{2}}\cos\left(ax\right)dx \cr
&= \Re\int_{0}^{\infty}e^{-x^{2}}e^{i\left|a\right|x}dx \cr
&= \Re\int_{0}^{\infty}\exp\left(-\left(x-\frac{i\left|a\right|}{2}\right)^{2}-\frac{\left|a\right|^{2}}{4}\right)dx \cr
&= \exp\left(-\frac{a^{2}}{4}\right)\Re\int_{0}^{\infty}\exp\left(-\left(x-\frac{i\left|a\right|}{2}\right)^{2}\right)dx \cr
&= \exp\left(-\frac{a^{2}}{4}\right)\Re\int_{-i|a|/2}^{\infty-i|a|/2}e^{-x^{2}}dx \cr
&= \exp\left(-\frac{a^{2}}{4}\right)\int_{0}^{\infty}e^{-x^{2}}dx. \cr
}
$$
Let $f(z)=e^{-z^2}$. We shall use your idea of constructing a rectangle:
$$C := \left\{z \in \mathbb{C} : \Re(z) \in [0,R] \text{ } \land \text{ } \Im(z) \in [0,a] \right\},$$
provided $R > a$.
Fortunately, we know $f(z)$ is entire. Using Cauchy's Residue Theorem, we can rewrite $\displaystyle \oint_{C}f(z)dz$ as
$$0 = \int_{0}^{R}f\left(z\right)dz + \int_{R}^{R+ia}f\left(z\right)dz + \int_{R+ia}^{ia}f\left(z\right)dz + \int_{ia}^{0}f\left(z\right)dz$$
$$\implies \int_{0}^{R}f\left(z\right)dz = \int_{R+ia}^{R}f\left(z\right)dz + \int_{ia}^{R+ia}f\left(z\right)dz + \int_{0}^{ia}f\left(z\right)dz.$$
Let the three integrals on the right side of the equality be $I_1$, $I_2$, and $I_3$ (from left to right). We will solve each of them separately as $R \to \infty$.
Let $z=R+iy$. Then bounding $|I_2|$, we get
$$
\eqalign{
\left|\int_{R}^{R+ia}e^{-z^{2}}dz\right| &= \left|\int_{0}^{a}\exp\left(-\left(R+iy\right)^{2}\right)idy\right| \cr
&\leq \int_{0}^{a}\left|\exp\left(-\left(R+iy\right)^{2}\right)\right|dy \cr
&= \int_{0}^{a}\frac{\exp\left(y^{2}\right)}{\exp\left(R^{2}\right)}dy. \cr
}
$$
Since $R>a$, we can take $R\to\infty$ so that $\displaystyle \int_{0}^{a}\frac{\exp\left(y^{2}\right)}{\exp\left(R^{2}\right)}dy \to 0$. By the Squeeze Theorem, we can conclude $I_2 \to 0$.
To evaluate $I_4$, we can use the definition of the Error Function to get
$$\int_{0}^{ia}e^{-z^{2}}dz = \frac{i\sqrt{\pi}}{2} \operatorname{erfi}(a).$$
To evaluate $I_3$, let us use your ideas. If $z = x+ia$, then
$$
\eqalign{
\int_{ia}^{R+ia}e^{-z^{2}}dz &= \int_{0}^{R}\exp\left(-\left(x+ia\right)^{2}\right)dx \cr
&= e^{a^{2}}\int_{0}^{R}e^{-x^{2}}e^{i\left(-2ax\right)}dx \text{ (Can you see your mistake?)} \cr
&= e^{a^{2}}\int_{0}^{R}e^{-x^{2}}\cos\left(2ax\right)dx-ie^{a^{2}}\int_{0}^{R}e^{-x^{2}}\sin\left(2ax\right)dx. \cr
}
$$
Ending up with $\displaystyle \int_{0}^{\infty}\exp\left(-x^{2}\right)\cos\left(2ax\right)dx$ when we have been trying to evaluate $\displaystyle \int_{0}^{\infty}\exp\left(-x^{2}\right)\cos\left(ax\right)dx$ from the get-go is somewhat circular (for lack of a better term), but I suppose we can use your link you provided to prove $\displaystyle\int_{0}^{\infty}\exp\left(-x^{2}\right)\cos\left(2ax\right)dx = \dfrac{\sqrt{\pi}}{2}e^{-a^{2}}$.
(Answer) We will prove
$$\int_{0}^{\infty}e^{-x^{2}}\sin\left(2ax\right)dx = \frac{\sqrt{\pi}}{2\exp\left(a^{2}\right)} \operatorname{erfi}(a).$$
Proof. Let $I(b) = \int_{0}^{\infty}e^{-x^{2}}\sin\left(bx\right)dx.$ Using the Leibniz Integral Rule, we differentiate with respect to $b$ to get
$$
\eqalign{
I'(b) &= \int_{0}^{\infty}\frac{\partial}{\partial b}e^{-x^{2}}\sin\left(bx\right)dx \cr
&= \int_{0}^{\infty}xe^{-x^{2}}\cos\left(bx\right)dx \cr
&= \left[-\frac{1}{2}e^{-x^{2}}\cos\left(bx\right)\right]_{0}^{\infty}-\frac{b}{2}\int_{0}^{\infty}e^{-x^{2}}\sin\left(bx\right) \cr
I'(b) + \frac{b}{2} I(b) &= \frac{1}{2}\cr
e^{b^2/4}\left(I'(b) + \frac{b}{2} I(b)\right)&= \frac{1}{2}e^{b^2/4}\cr
\frac{d}{db}\left(e^{b^2/4} I\left(b\right)\right) &= \frac{1}{2}e^{b^2/4} \cr
\int \frac{d}{db}\left(e^{b^2/4} I\left(b\right)\right)db &= \int \frac{1}{2}e^{b^2/4}db \cr
e^{b^2/4} I(b)&= \int e^{u^2}du\cr
&= \frac{\sqrt{\pi}}{2} \operatorname{erfi}\left(\frac{b}{2}\right) + C.
}
$$
By transitivity, we have
$$I(b) = e^{-b^2/4}\left(\frac{\sqrt{\pi}}{2} \operatorname{erfi}\left(\frac{b}{2}\right) + C\right) = \int_{0}^{\infty}e^{-x^{2}}\sin\left(bx\right)dx.$$
To make $C$ vanish, we let $x=0$. This means
$$I(0) = C = 0.$$
Thus,
$$I(b) = e^{-b^2/4}\frac{\sqrt{\pi}}{2} \operatorname{erfi}\left(\frac{b}{2}\right).$$
Substituting $2a$, we get $I(2a)$ to be
$$\int_{0}^{\infty}e^{-x^{2}}\sin\left(2ax\right)dx = \frac{\sqrt{\pi}}{2\exp\left(a^{2}\right)} \operatorname{erfi}(a).$$
Q.E.D.
Putting everything together, letting $R \to \infty$, and multiplying by $\exp\left(-\frac{a^{2}}{4}\right)$, we get
$$\exp\left(-\frac{a^{2}}{4}\right)\int_{0}^{\infty}e^{-x^{2}}dx = \exp\left(-\frac{a^{2}}{4}\right)\left(0 +e^{a^2}\dfrac{\sqrt{\pi}}{2}e^{-a^{2}} -ie^{a^2}\frac{\sqrt{\pi}}{2\exp\left(a^{2}\right)} \operatorname{erfi}(a) + \frac{i\sqrt{\pi}}{2} \operatorname{erfi}(a) \right).$$
In conclusion, our integral $I$ is
$$\int_{0}^{\infty}e^{-x^{2}}\cos\left(ax\right)dx = \frac{\sqrt{\pi}}{2}\exp\left(-\frac{a^{2}}{4}\right).$$
Please let me know if there are any questions.
