I am going try to reproduce your answer (at least how I think that you got there), so we can see where it went wrong.
Let
$$ x=r\cos\theta, \quad y = r\sin\theta. $$
Then
$$ z^2 = 3(x^2+y^2) = 3r^2 \quad \Rightarrow \quad z = r\sqrt{3} $$
where the final implication follows since both $z$ and $r$ are non-negative.
Hence
$$ \Psi(r, \theta) = (r\cos \theta, r\sin\theta, r\sqrt{3}), $$
parameterizes our surface, but we must also find the correct intervals for $r$ and $\theta$. First, we have
$$ 0 < z < 3 \quad \Leftrightarrow \quad 0 < r\sqrt{3} < 3 \quad \Leftrightarrow \quad 0 < r < \sqrt{3}. $$
Second, we know that $x,y > 0,$ so that they lie in the first quadrant, and we can conclude that $0 < \theta < \frac{\pi}{2}$.
Now it is only left to compute the surface element $dA$ before we can attack the integral, which can be done via
$$ dA = \left|\frac{\partial \Psi}{\partial r} \times \frac{\partial \Psi}{\partial \theta}\right| \, dr\,d\theta. $$
The formula $\sqrt{1+f_x^2+f_y^2}$ that you seem to have used works only if you have a parameterization of the form $(x,y,f(x,y)),$ which is not the case in our case here.
We have
$$ \frac{\partial \Psi}{\partial r} = (\cos\theta, \sin\theta, \sqrt{3}), $$
and
$$ \frac{\partial \Psi}{\partial \theta} = (-r\sin\theta, r\cos\theta, 0), $$
so
\begin{align}
\frac{\partial \Psi}{\partial r} \times \frac{\partial \Psi}{\partial \theta}
&= \begin{vmatrix}
e_1 & e_2 & e_3\\
\cos\theta & \sin\theta & \sqrt{3}\\
-r\sin\theta & r\cos\theta & 0
\end{vmatrix}\\[0.2cm]
&= e_1(0-\sqrt{3}r\cos\theta) + e_2(-\sqrt{3}r\sin\theta-0) + e_3(r\cos^2\theta+r\sin^2\theta)\\[0.1cm]
&= (-\sqrt{3}r\cos\theta, -\sqrt{3}r\sin\theta, r),
\end{align}
and thus
\begin{align}
\left|\frac{\partial \Psi}{\partial r} \times \frac{\partial \Psi}{\partial \theta}\right| &= \sqrt{(-\sqrt{3}r\cos\theta)^2 + (-\sqrt{3}r\sin\theta)^2+r^2}\\
&=\sqrt{3r^2\cos^2\theta+3r^2\sin^2\theta+r^2} = r\sqrt{4} = 2r.
\end{align}
We have thus found your error - your surface element was wrong!
To complete the answer, we have
\begin{align}
\iint_S x^2+y^2 \, dA &=
\int_0^{\pi/2}\int_0^{\sqrt{3}} r^2 \cdot 2r \, dr\, d\theta
= \pi \int_0^{\sqrt{3}} r^3 \, dr
= \frac{\pi}{4} [r^4]_0^{\sqrt{3}} = \frac{9\pi}{4}
\end{align}