$$\int (6x^2-2)^{\frac{3}{2}} \mathrm{d}x$$
Tried converting to trigonometric functions using substitution $6x^2-2 = t^2$ and then $t^2 = 2\tan \theta$, but I get an equation in $\sec \theta$ with higher powers like $ \int \sec^5 \theta $ etc. How do I solve these or the original problem, any hints would be helpful. Thanks.
Rewrite the integral as,
$$I=\int (6x^2-2)^{\frac{3}{2}} dx =\sqrt{\frac 83} \int [(\sqrt 3 x)^2-1]^{3/2}d(\sqrt 3 x)$$
and let $\sqrt 3 x=\cosh t$,
$$I=\sqrt{\frac 83} \int \sinh^4 t dt$$
Expend the integrand with the identities $2\sinh^2 t = \cosh 2t -1 $ and $2\cosh^2 2t = \cosh 4t +1 $,
$$\sinh^4 t = \frac 38 - \frac 12 \cosh 2t +\frac 18 \cosh 4t$$
Then, integrate,
$$I=\sqrt{\frac 23}\left(\frac 34 t -\frac12 \sinh 2t+ \frac{1}{16}\sinh 4t\right)+C$$
Note that the substitution with $\cosh t$ results in economical integration afterwards.
The standard reduction formula for $$\int \sec^n \theta \,d\theta , \qquad n \geq 3,$$ is $$\boxed{\int \sec^n \theta \,d\theta = \frac{1}{n-1} \sec^{n-2} \theta \tan \theta + \frac{n-2}{n-1} \int \sec^{n - 2} \theta \,d\theta} .$$ (See this question for more information about this formula, and see either of its answers for a derivation.)
Substituting $n = 5$ then gives $$\int \sec^5 \theta \,d\theta = \frac{1}{4}\sec^3 \theta \tan \theta + \frac{3}{4} \int \sec^3 \theta \,d\theta .$$ Applying the same rule to $\int \sec^3 \theta \,d\theta$ gives an expression for $\int \sec^5 \theta \,d\theta$ in terms of $$\int \sec \theta \,d\theta$$ and closed-form expressions, and that latter integral can be evaluated in many ways.
First of all take $2^{\frac{3}{2}}$ out of the expression
Then use $x=\frac{\sec(u)}{\sqrt{3}}$
In the end you'll get
$\frac{2^{\frac{3}{2}}}{\sqrt{3}}\int \sec(u) {\tan (u)}^4 \ du$
If you solve this and substitute back you'll get
$$\dfrac{\sqrt{3}\ln\left(\left|\sqrt{3x^2-1}+\sqrt{3}x\right|\right)+x\sqrt{3x^2-1}\left(6x^2-5\right)}{2^\frac{3}{2}}$$
