Note that we can evaluate the integral in Cartesian coordinates directly. We proceed by writing
$$\begin{align}
\int_{-\infty}^\infty\int_{-\infty}^\infty e^{-x^2-y^2-axy}\,dx\,dy&=\int_{-\infty}^\infty e^{-y^2(1-a^2/4)}\left(\int_{-\infty}^\infty e^{-(x+ay/2)^2}\,dx\right)\,dy\\\\
&=\int_{-\infty}^\infty e^{-y^2(1-a^2/4)}\left(\int_{-\infty}^\infty e^{-x^2}\,dx\right)\,dy\\\\
&=\left(\int_{-\infty}^\infty e^{-x^2}\,dx\right)\left(\int_{-\infty}^\infty e^{-y^2(1-a^2/4)}\,dy\right)\\\\
&=\sqrt \pi \left(\int_{-\infty}^\infty e^{-y^2(1-a^2/4)}\,dy\right)\\\\
&=\sqrt \pi \frac{1}{\sqrt{1-a^2/4}}\int_{-\infty}^\infty e^{-y^2}\,dy\\\\
&=\frac{2\pi}{\sqrt{4-a^2}}
\end{align}$$
If one prefers to transform to polar coordinates, then we can write
$$\begin{align}
\int_{-\infty}^\infty\int_{-\infty}^\infty e^{-x^2-y^2-axy}\,dx\,dy&=\int_{0}^{2\pi}\int_{0}^\infty \rho e^{-\rho^2(1+a\sin(2\phi)/2)}\,d\rho\,d\phi\\\\
&=\int_{0}^{2\pi} \frac{1}{2+a\sin(2\phi)}\,d\phi\\\\
&=\frac12 \int_{0}^{4\pi} \frac{1}{2+a\sin(\phi)}\,d\phi\\\\
&=\frac12 \left(\int_{0}^{\pi} \frac{1}{2+a\sin(\phi)}\,d\phi+\int_{\pi}^{2\pi} \frac{1}{2+a\sin(\phi)}\,d\\\\+\int_{2\pi}^{3\pi} \frac{1}{2+a\sin(\phi)}\,d\phi+\int_{3\pi}^{4\pi} \frac{1}{2+a\sin(\phi)}\,d\phi\right)\\\\
&=\int_{0}^{\pi} \frac{1}{2+a\sin(\phi)}\,d\phi+\int_{0}^{\pi} \frac{1}{2-a\sin(\phi)}\,d\phi
\end{align}$$
The integrals $\int_{0}^{\pi} \frac{1}{2+a\sin(\phi)}\,d\phi$ and $\int_{0}^{\pi} \frac{1}{2-a\sin(\phi)}\,d\phi$ can be evaluated using either contour integration or the tangent half-angle substitution with the result giving the expected result.