Using double angle formula, the integrand can be rewritten as
\begin{equation}
I=\int_0^{\pi/2} \frac{x^5}{2-\cos^2(x)}\ dx=\int_0^{\pi/2} \frac{2x^5}{3-\cos(2x)}\ dx
\end{equation}
Mapping the variable $2x\mapsto x$, we have
\begin{equation}
I=\frac{1}{32}\int_0^{\pi} \frac{x^5}{3-\cos x}\ dx
\end{equation}
Using identity (proof can be seen here)
\begin{equation}
1+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\cos(n x)=\frac{a^2-b^2}{a^2+b^2-2ab\cos x}\qquad,\qquad\mbox{for}\, |b|<a
\end{equation}
and the correspondence values $a=\dfrac{2+\sqrt{2}}{2}$ and $b=\dfrac{2-\sqrt{2}}{2}$, one may find
\begin{equation}
1+2\sum_{n=1}^\infty \left(3-2\sqrt{2}\right)^n\cos(n x)=\frac{2\sqrt{2}}{3-\cos x}
\end{equation}
Therefore
\begin{align}
I&=\frac{1}{64\sqrt{2}}\int_0^{\pi} \left[x^5+2\sum_{n=1}^\infty \left(3-2\sqrt{2}\right)^n x^5\cos(n x)\right]\ dx\\
&=\frac{1}{64\sqrt{2}} \left[\frac{\pi^6}{6}+2\sum_{n=1}^\infty \left(3-2\sqrt{2}\right)^n \int_0^{\pi} x^5 \cos(n x)\ dx\right]\\
\end{align}
The rest part can be done by multiple times integration by parts and using $\sin(n\pi)=0$ for $n\in\mathbb{Z}$. We will obtain
\begin{equation}
I=\frac{\pi^6 \sqrt{2}}{768}+\frac{\sqrt{2}}{64}\sum_{n=1}^\infty \left(3-2\sqrt{2}\right)^n \left[\frac{5\pi^4\cos(\pi n)}{n^2}-\frac{60\pi^2\cos(\pi n)}{n^4}+\frac{120\cos(\pi n)}{n^6}-\frac{120}{n^6}\right]\
\end{equation}
Using
\begin{equation}
\cos(n\pi)=\begin{cases}\,\,+1&,\,\,\mbox{if}\,\, n\,\,\mbox{is even}\\[12pt]
\,\,-1&,\,\,\mbox{if}\,\, n\,\,\mbox{is odd}\\
\end{cases}
\end{equation}
and the representation of polylogarithm function in term of its infinite series, we finally obtain
\begin{align*}
\int_0^{\pi/2} \frac{x^5}{2-\cos^2(x)}\ dx=&\,\frac{\pi^6 \sqrt{2}}{768}+\frac{5 \sqrt{2}\pi^4}{64}\text{Li}_2\left(2\sqrt2-3\right)-\frac{15\sqrt{2}\pi^2}{16}\text{Li}_4\left(2\sqrt2-3\right)\\
&+\,\frac{15\sqrt{2}}{8}\bigg[\text{Li}_6\left(2\sqrt2-3\right)-\text{Li}_6\left(3-2\sqrt2\right)\bigg]
\end{align*}
The same approach can be applied for evaluating the second integral.
