Let's recall (and modify) one of the known approaches to get the first of these integrals.
Assume (temporarily) that $0<\Re a<\Re b$. Let $\gamma_+$ be the boundary of $$\{z:|z|<1,0<\arg z<\pi/2\}$$ (consisting of the segment $[0,1]$ of the real line, the quartercircle $\{e^{ix}:0\leqslant x\leqslant\pi/2\}$ and the segment $[i,0]$ of the imaginary line), and similarly $\gamma_-$ be the boundary of $$\{z:|z|<1,-\pi/2<\arg z<0\}.$$ Then, by Cauchy's integral theorem (the singularity at $z=0$ taken as a limit)
$$0=\int_{\gamma_+}z^{b-a}(1+z^2)^{a-1}~dz=\int_{\gamma_-}z^{b-a}(1+z^2)^{a-1}~dz.$$
Writing explicitly the integrals along each piece of the contours, we get
$$0=F_+(a,b)+2^{a-1}i\int_0^{\pi/2}e^{ibx}\cos^{a-1}x~dx-ie^{i(b-a)\pi/2}F_-(a,b),\\0=F_+(a,b)-2^{a-1}i\int_0^{\pi/2}e^{-ibx}\cos^{a-1}x~dx+ie^{i(a-b)\pi/2}F_-(a,b),\\\text{where }F_\pm(a,b):=\int_0^1 x^{b-a}(1\pm x^2)^{a-1}~dx.$$
Observe that $F_-(a,b)=\frac{1}{2}\mathrm{B}\left(a,\frac{b-a+1}{2}\right)$. Thus, taking the difference of the preceding equalities, we get rid of $F_+$, and using the reflection formula for $\Gamma\left(\frac{b-a+1}{2}\right)$, we obtain the first formula in the question (which then holds, by analytic continuation, for any $a,b$ with $\Re a>0$). Taking the sum instead, we get $$2^{a-1}\int_0^{\pi/2}\cos^{a-1}x\sin bx~dx=F_+(a,b)+F_-(a,b)\sin\frac{(b-a)\pi}{2}.$$ This time, the $F_+$ is retained, and there's no simple formula for this integral. (Of course, $F_+(a,b)$ can be expressed in terms of incomplete beta functions, or evaluated in closed form in some particular cases, say when $a$ and $b$ are both (half-)integers, or when $(b-a-1)/2$ is an integer.)
The upper limit of $\pi$ makes it a different story. For $0<\Re a<\Re b$, integrating $z^{b-a}(1-z^2)^{a-1}$ over the boundary of $\{z : |z|<1,0<\arg z<\pi\}$ (and using the reflection formula for $\Gamma$ again), we get $$\int_0^\pi e^{ibx}\sin^{a-1}x~dx=\frac{\pi e^{i\pi b/2}}{2^{a-1}a\mathrm{B}\left(\frac{a+b+1}{2},\frac{a-b+1}{2}\right)}$$ (which, again by analytic continuation, holds for any $a,b$ with $\Re a>0$).