I don't know how to evaluate it. I know there is one method using the gamma function. BUT I want to know the solution using a calculus method like polar coordinates.
$$\int_{-\infty}^\infty x^2 e^{-x^2}\mathrm dx$$
I will wait for a solution. Thank you.
In order to solve the integral by polar coordinates first consider $I_s = \int_{-\infty}^\infty \mathrm{e}^{-s x^2} \mathrm{d} x$. The integral you seek will be obtained by differentiation as $-\left. \frac{\mathrm{d}}{\mathrm{d} s} I_s \right|_{s=1}$.
Now, to evaluate $I_s$:
$$ I_s^2 = \int_{-\infty}^\infty \mathrm{e}^{-s x^2} \mathrm{d} x \cdot \int_{-\infty}^\infty \mathrm{e}^{-s y^2} \mathrm{d} y = \int_{-\infty}^\infty \int_{-\infty}^\infty \mathrm{e}^{-s (x^2 + y^2)} \, \mathrm{d} x \mathrm{d} y $$ Now change variables into polar coordinates $x = r \sin \theta$ and $y = r \cos \theta$. $$ I_s^2 = \int_{0}^{2 \pi} \mathrm{d} \theta \int_0^\infty \mathrm{e}^{-s r^2} \cdot r \, \mathrm{d} r = \pi \int_0^\infty \mathrm{e}^{-s t} \mathrm{d} t = \frac{\pi}{s} $$ where $t = r^2$ change of variable has been made.
Now, since $I_s > 0$ for $s >0$, we obtain $I_s = \sqrt{\frac{\pi}{s}}$.
The integral in question now follows: $$ \int_{-\infty}^\infty x^2 \mathrm{e}^{-x^2} \mathrm{d} x = \left. -\frac{\mathrm{d}}{\mathrm{d} s} \sqrt{\frac{\pi}{s}} \right|_{s=1} = \left. \frac{\sqrt{\pi}}{2} s^{-\frac{3}{2}} \right|_{s=1} = \frac{\sqrt{\pi}}{2} $$
(As the OP wants a solution without using the gamma function.) Following Davide's suggestion, we write: $$ \int_{-\infty}^{\infty} x^2 e^{-x^2} dx = -\frac{1}{2}\int_{-\infty}^{\infty} x \cdot (-2x e^{-x^2}) dx. $$ Let $u = x$ and $v = e^{-x^2}$. We have $\frac{dv}{dx} = -2xe^{-x^2}$. Integrating by parts: $$ -\frac{1}{2}\int_{-\infty}^{\infty} u \frac{dv}{dx} dx = -\frac{1}{2} \left. uv \right|_{-\infty}^{\infty} + \frac{1}{2} \int_{-\infty}^{\infty} v \frac{du}{dx} dx. $$ I will leave it as an exercise to compute the first term. (Hint: it should come out to $0$. :)) The integral appearing in the second term (ignoring the factor of $\frac{1}{2}$ in the front) simplifies to: $$ \int_{-\infty}^{\infty} v \frac{du}{dx} dx = \int_{-\infty}^{\infty} e^{-x^2} dx. $$ This is the famous Gaussian integral, whose value is $\sqrt{\pi}$. You should now be able to evaluate integral easily.
Detailing Srivatsan Narayanan's solution. It is known that the functional equation of the gamma function may be derived applying the integration by parts technique. Its value at $1/2$ may be evaluated by computing a double integral over the first quadrant in Cartesian and polar coordinates. Let's apply similar ideas in this case. Let $f(x)=x^{2}e^{-x^{2}}$. Since $ f(-x)=f(x)$ the integral $\int_{-\infty }^{\infty }f(x)\mathrm{d}x=2\int_{0}^{\infty }f(x)\mathrm{d}x$. Integrating by parts $$ \int x^{2}e^{-x^{2}}\mathrm{d}x=\int x\cdot xe^{-x^{2}}\mathrm{d}x, $$ since $$ \int xe^{-x^{2}}\mathrm{d}x=-\frac{1}{2}e^{-x^{2}} $$ and $\frac{dx}{dx}=1$, we get $$\begin{eqnarray*} \int x^{2}e^{-x^{2}}\mathrm{d}x &=&x\left( -\frac{1}{2}e^{-x^{2}}\right) -\int -\frac{ 1}{2}e^{-x^{2}}\,\mathrm{d}x \\ &=&-\frac{1}{2}xe^{-x^{2}}+\frac{1}{2}\int e^{-x^{2}}\,\mathrm{d}x.\tag{0} \end{eqnarray*}$$
And so, $$\begin{eqnarray*} \int_{0}^{\infty }x^{2}e^{-x^{2}}dx &=&\left. -\frac{1}{2} xe^{-x^{2}}\right\vert _{0}^{\infty }+\frac{1}{2}\int_{0}^{\infty }e^{-x^{2}}\,\mathrm{d}x \\ &=&\left( \lim_{c\rightarrow \infty }-\frac{1}{2}ce^{-c^{2}}\right) +\frac{1 }{2}0e^{-0^{2}}+\frac{1}{2}\int_{0}^{\infty }e^{-x^{2}}\,\mathrm{d}x \\ &=&0+0+\frac{1}{2}\int_{0}^{\infty }e^{-x^{2}}\,\mathrm{d}x \\ &=&\frac{1}{2}\int_{0}^{\infty }e^{-x^{2}}\,\mathrm{d}x.\tag{1} \end{eqnarray*}$$
Consequently, $$ I:=\int_{-\infty }^{\infty }x^{2}e^{-x^{2}}\mathrm{d}x=2\int_{0}^{\infty }x^{2}e^{-x^{2}}\mathrm{d}x=\int_{0}^{\infty }e^{-x^{2}}\,\mathrm{d}x.\tag{2} $$
To evaluate this last integral we compute the following double integral in Cartesian and polar coordinates ($r^{2}=x^{2}+y^{2}$, $x=r\cos \theta ,y=r\sin \theta $). Since the Jacobian of the transformation $\frac{\partial (x,y)}{\partial (r,\theta )}=r$, we have $$ \int_{x=0}^{\infty }\int_{y=0}^{\infty }e^{-x^{2}-y^{2}}\mathrm{d}x\mathrm{d}y=\int_{\theta =0}^{\pi /2}\int_{r=0}^{\infty }e^{-r^{2}}r\mathrm{d}r\mathrm{d}\theta.\tag{3} $$ Comparing the LHS $$ \int_{x=0}^{\infty }\int_{y=0}^{\infty }e^{-x^{2}-y^{2}}\mathrm{d}x\mathrm{d}y=\left( \int_{0}^{\infty }e^{-x^{2}}\mathrm{d}x\right) \left( \int_{0}^{\infty }e^{-y^{2}}\mathrm{d}y\right) =I^{2}\tag{4} $$ with the RHS $$\begin{eqnarray*} I^2 &=&\int_{\theta =0}^{\pi /2}\int_{r=0}^{\infty }e^{-r^{2}}r\mathrm{d}r\mathrm{d}\theta = \frac{\pi }{2}\int_{0}^{\infty }e^{-r^{2}}r\mathrm{d}r \\ &=&\frac{\pi }{2}\left. \left( -\frac{1}{2}e^{-r^{2}}\right) \right\vert _{0}^{\infty }=\frac{\pi }{2}\left( \lim_{c\rightarrow \infty }-\frac{1}{2} e^{-c^{2}}+\frac{1}{2}e^{-0^{2}}\right) \\ &=&\frac{\pi }{2}\left( 0+\frac{1}{2}\right) =\frac{\pi }{4},\tag{5} \end{eqnarray*}$$ yields $$ I=\frac{\sqrt{\pi }}{2}.\tag{6} $$
First, since the integrand is symmetric around $0$, we can write it as twice the integral from $0$ to $\infty$. Now, change variables by letting $u=x^2$ so that $du=2xdx$. Then our integral becomes $$\int_{-\infty}^\infty x^2e^{-x^2} dx=\int_{0}^\infty xe^{-x^2} 2xdx=\int_{0}^\infty u^{\frac{1}{2}} e^{-u}du=\Gamma\left(\frac{3}{2}\right) =\frac{\sqrt{\pi}}{2}$$ by the definition of the Gamma function along with the fact that $\Gamma(1/2)=\sqrt{\pi}$. (7 proofs of this last identity, or equivalently the identity $\int_{-\infty}^\infty e^{-x^2}dx =\sqrt{\pi}$ are given on this Math Stack Exchange post.)
If you know that $\int_{-\infty}^\infty e^{-x^2}=\sqrt\pi$, then you can use it to easily solve this.
Differentiating $e^{-x^2}$ twice,
$$\frac{d^2}{dx^2}e^{-x^2}=-2e^{-x^2}+4x^2e^{-x^2}$$
Now integral of the left hand side is $0$, as $\int\left(\frac{d^2}{dx^2}e^{-x^2}\right)dx=\frac{d^2}{dx^2}\int e^{-x^2}dx=\frac{d^2}{dx^2}\sqrt\pi=0$.
So integrating, we get
$$\begin{array}{rcl} 0&=&\int -2e^{-x^2}+4x^2e^{-x^2}\,dx\\ &=&-2\sqrt\pi+4\int x^2e^{-x^2}\,dx\\ \implies\int x^2e^{-x^2}\,dx&=&\frac{\sqrt\pi}{2} \end{array} $$
