Here is an alternative approach, but only a start. At some point i had to give up.
The idea was to use complex analysis, bring the computations as far as possible, then try to connect the residues obtained to some "known expressions". Unfortunately, there are no "known expressions" (from my view), have to stop there. Since there was still some work done, i decided to submit, sometimes also unsuccessful trials have their merits.
Let us consider the functions $f:\Bbb R\to\Bbb R$, $g:\Bbb C-\{0,\pm s,\pm s\}$,
$$
\begin{aligned}
f(t) &= \frac {\cos(A\cos t)}{a^2\sin^2 t+b^2 \cos ^2 t}\ ,\qquad a,b>0\ ,\ A\in\Bbb R\ ,\\
g(z) &= \frac
{\displaystyle\cos\left(\frac A2\left(z+\frac 1z\right)\right)}
{\displaystyle a^2\left(\frac 1{2i}\left(z-\frac 1z\right)\right)^2
+ b^2\left(\frac 12 \left(z+\frac 1z\right)\right)^2}
\cdot\frac 1{iz}\ ,
\end{aligned}
$$
for some pole value $s$ to be specified in the sequel.
Now observe the relation $f(t)=f(\pm t)=f(\pi\pm t)$,
and we have then for the value $J$ of the integral to be calculated
$$
\begin{aligned}
J&=
J(A;a,b)\\
&=
\int_0^{\pi/2}
f(t)\; dt
=
\frac 14\int_0^{2\pi}
f(t)\; dt
\\
&=
\frac 14\int_0^{2\pi}
g(e^{it})\; i\ e^{it}\;dt
\\
&=
\frac 14\int_{\text{Circle $C$ around $0$ with radius $1$}}
g(z)\; dz
\\
&=2\pi i\sum_{a\text{ singularity of $g$ inside $C$}}\operatorname{Res}_{z=a}g(z)\ .
\end{aligned}
$$
Time to compute explicitly the singular points. First, we consider the zeros of the
"denominator" of $g$, as it was written,
$$
\begin{aligned}
&\left(
a^2\left(\frac 1{2i}\left(z-\frac 1z\right)\right)^2
+
b^2\left(\frac 12 \left(z+\frac 1z\right)\right)^2
\right)
\cdot
z
\\
&\qquad=
\frac z4
\cdot
\left( (b+a)z+(b-a)\frac 1z \right)
\left( (b-a)z+(b+a)\frac 1z \right)
\\
&\qquad=
\frac 14\cdot\frac 1z
\cdot
\Big(\ (b+a)z^2+(b-a) \ \Big)
\Big(\ (b-a)z^2+(b+a) \ \Big)
\end{aligned}
$$
In case $a\ne b$, only the factor $\Big(\ (b+a)z^2+(b-a) \ \Big)$
introduces poles of $g$ inside $C$, the already mentioned ones,
$$
\pm s=\pm i \sqrt{\frac{b-a}{b+a}}
\ .
$$
The other factor introduces poles outside the unit circle $C$, they do not contribute to the integral.
It is maybe good to cover first the case $a=b$ in detail, then the general case along the same lines.
In both cases, the complication comes from the essential singularity in zero.
So let us compute first the integral in case $a=b=1$.
$$
\begin{aligned}
J&=
J(A;1,1)\\
&=
\frac 14\int_C\cos\left(\frac A2\left(z+\frac 1z\right)\right)\cdot\frac 1{iz}\; dz
\\
&=2\pi i\frac 14\operatorname{Res}_{z=0}
\cos\left(\frac A2\left(z+\frac 1z\right)\right)\cdot\frac 1{iz}
\\
&=\frac \pi 2
\sum_{k\ge 0}(-1)^k\frac 1{(2k)!}
\left(\frac A2\right)^{2k}\binom{2k}k
=
\\
&=
\frac \pi 2
\sum_{k\ge 0}(-1)^k\left(\frac A2\right)^{2k}\frac 1{k!^2}\ .
\end{aligned}
$$
Let us check the above numerically for $A=1,2,3,4$.
{for( A=1, 4,
J = intnum(t=0, Pi/2, cos(A*cos(t)));
S = Pi/2*suminf( k=0, (-1)^k*(A/2.)^(2*k)/(k!)^2 );
print("A=", A, "\nJ(A) ~ ", J, "\nS ~ ", S, "\n"); )}
A=1
J(A) ~ 1.2019697153172064991366624462957556118924127364894
S ~ 1.2019697153172064991366624462957556118924127364894
A=2
J(A) ~ 0.35168681347830044589240089314016748939896601821583
S ~ 0.35168681347830044589240089314016748939896601821583
A=3
J(A) ~ -0.40848865553578915389261538954946675593562657336797
S ~ -0.40848865553578915389261538954946675593562657336797
A=4
J(A) ~ -0.62384146252142305380166549056420011118837803881275
S ~ -0.62384146252142305380166549056420011118837803881275
Pari/GP provides each time the same (for the given precision).
The general case now.
The residues in $\pm s$ are simple, we calculate them as
$$
2\pi\cdot \frac 14
\cos\left(\frac A2\left(z+\frac 1z\right)\right)
\cdot\frac 1{b^2-a^2}
\cdot\frac z{(z-s)(z+s)\left(z-\frac 1s\right)\left(z+\frac 1s\right)}\ .
$$
The values coincide, are each
$$
2\pi\cdot \frac 14
\cos\left(\frac A2\left(s+\frac 1s\right)\right)
\cdot\frac s{2s\left(s-\frac 1s\right)\left(s+\frac 1s\right)}\ .
$$
The above may be simplified. But let us face the real problem, the essential singularity in zero. We rewrite
$$
g(z)
=
\frac
{\displaystyle\cos\left(\frac A2\left(z+\frac 1z\right)\right)}
{\displaystyle
(b+a)(b-a)
\Big(\ 1+\frac{b-a}{b+a}z^2 \ \Big)
\Big(\ 1+\frac{b+a}{b-a}z^2 \ \Big)
}
\cdot\frac 4{i}
\cdot z
\ .
$$
So (after a partial fraction decomposition) we need to compute the residue in $0$ of functions of the shape
$$
\begin{aligned}
h(z)=h_c(z)
&=
\frac
{\displaystyle\cos\left(\frac A2\left(z+\frac 1z\right)\right)}
{1-cz^2}
\cdot z
\\
&=
\cos\left(\frac A2\left(z+\frac 1z\right)\right)
(z+c z^3+c^2z^5+\dots)
\\
&=
\left(
\sum_{k\ge 0}
(-1)^k\frac 1 {(2k)!}
\left(\frac A2\right)^{2k}
\sum_{p+q=2k}\binom {2k}{p\; q} z^{p-q}
\right)
\left(
\sum_{n\ge 1}
c^n\, z^{2n-1}
\right)
\ ,
\end{aligned}
$$
we isolate the part of the above, considered as a formal series,
in $z^{-1}$,
so first of all we need $p-q=2n\le -2$,
so from $p+q=2k$, $p-q=-2n$, we get $p=k-n$, $q=k+n$,
so the residue is in terms of $k,n$
$$
\sum_{k\ge n\ge 1}
(-1)^k\frac 1 {(k+n)!(k-n)!}
\left(\frac A2\right)^{2k}
c^n\ ,
$$
or in terms of $p,q$ constrained to have the same parity:
$$
\sum_{\substack{q\ge p\ge 0\\p-q\equiv0[2]}}
(-1)^{(p+q)/2}\frac 1 {p!q!}
\left(\frac A2\right)^{p+q}
c^{(q-p)/2}\ ,
$$
At this point i decided to stop and submit. Using computers, we can evaluate, but there is no "closed fromula".
