\begin{align*} \int \sec x dx & = \int {{\cos^2x} \over {\cos^3x}}dx \\&= \int \sec^3x \cos^2x dx =\int \sec^3x (1-\sin^2x) dx \\&=\int \sec^2x \sec x dx -\int \tan^2x \sec x dx \\&=\tan x \sec x + \int \tan^2x \sec x dx -\int \tan^2x \sec x dx \\&=\tan x \sec x. \end{align*} I kind of messed it up. I know how to derive the $\int \sec x dx$ with alternative way so I can find out it is wrong but can't find out why.
Will be very appreciated with your help.
\begin{align*} \int \sec x dx & = \int {{\cos^2x} \over {\cos^3x}}dx \\&= \int \sec^3x \cos^2x dx =\int \sec^3x (1-\sin^2x) dx \\&=\int \sec^2x \sec x dx -\int \tan^2x \sec x dx \\&=[\tan x \sec x - \int \tan^2x \sec x dx] -\int \tan^2x \sec x dx \\-\int \sec x dx &= \tan x \sec x -2\int \sec^3x dx \end{align*} but to calculate $\int \sec^3x$ you must know what is the $\int \sec x dx$ \begin{align*} \\-\int \sec x dx &= \tan x \sec x - (\tan x \sec x + \int \sec x dx) \\&=-\ln|\sec x +\tan x| \\\int \sec x dx &= \ln|\sec x +\tan x|. \end{align*}
You made a sign error when you integrated by parts. You correctly obtained \begin{align*} \int \sec x~dx & = \int \frac{\cos^2x}{\cos^3x}~dx\\ & = \int \sec^3x\cos^2x~dx\\ & = \int \sec^3x(1 - \sin^2x)~dx\\ & = \int \sec^3x~dx - \int \sec^3x\sin^2x~dx\\ & = \int \sec^2x\sec x~dx - \int \tan^2x\sec x~dx \end{align*} To do the first integral, we integrate by parts. Setting $u = \sec x$ and $dv = \sec^2x~dx$ yields \begin{align*} u & = \sec x & dv & = \sec^2x~dx\\ du & = \sec x\tan x & v & = \tan x \end{align*} Hence, $$\int \sec^2x\sec x~dx = uv - \int v~du = \sec x\tan x - \int \tan^2x\sec x~dx$$ so you should have obtained $$\int \sec x~dx = \sec x\tan x - 2\int \tan^2x\sec x~dx$$
The answers to this question describe many clever ways of evaluating the integral $$\int \sec x~dx$$
