Too Long for a comment. (see edit, it now contains the closed form.)
This is equivalent to a one variable function.
Substitute $x\to x^2$, and then $x=\frac1{a}\sin(t)$ to get
$$ \int_0^1 \frac{\ln^2(x)}{\sqrt{x(1-a^2x)}}dx=8\int_0^1 \frac{\ln^2(x)}{\sqrt{1-a^2x^2}}dx=\frac{8}{a}\int_0^{\sin^{-1}(a)}\ln^2\left(\frac{\sin t}{a}\right)dt
\\=\frac{8\ln^2(a)\sin^{-1}(a)}{a}-\frac{16\ln(a)}{a}\int_0^{\sin^{-1}(a)}\ln\sin x\,\,dx+\frac{8}{a}\int_0^{\sin^{-1}(a)}\ln^2\sin x\,\,dx$$
For certain algebraic $a$'s ,I think we can find a closed form.
For example, the case $a=\frac{\sqrt{2}}{2}$ corresponds to $\mathcal{I}(2,1)$,
and using the (wonderful) results obtained by @RandomVariable, we have
$$\mathcal{I}(2,1)=\frac{\pi}{2}\ln^2(2)-4\ln(2)(G+\frac{\pi}{2}\ln2)+8\left(\frac{\pi^{3}}{192} + G\frac{ \log(2)}{2} + \frac{3 \pi}{16} \log^{2}(2) - \text{Im} \ \text{Li}_{3}(1-i)\right)\,\,\left(=\frac{\pi^3}{24}-8\Im\operatorname{Li_3}(1-i)\right)$$
Edit
After some work I've been able to find the closed form. Towards the end, there's a huge cancellation which bothers me- there must be a straightforward way, but I'm blind.
I'll sketch how I found it: consider $\displaystyle \int_0^1 \frac{\ln^2(x)}{\sqrt{x(1-\sin(\pi\theta)^2x)}}dx=\frac{8}{\sin(\pi\theta)}\int_0^{\pi\theta} \ln^2\left(\frac{\sin(x)}{\sin(\pi\theta)}\right)dx\tag{1}$
Since $\displaystyle \ln(\sin x)=-\ln2-\sum_{n=1}^{\infty} \frac{\cos(2nx)}{n}$ we have
$\displaystyle \int_0^{\pi\theta} \ln(\sin x)dx=-\pi\theta\ln2-\frac12\operatorname{Cl_2}(2\pi\theta)\tag{2}$.
(And, as was done in @RandomVariable's answer linked above,) since $\displaystyle \Re\ln^2(1-e^{i2\pi x})=\ln^2(2\sin(\pi x))-(\frac{\pi}{2}-x)^2$, we have
$$\int_0^{\pi\theta} \ln^2(\sin x)dx\\=\int_0^{\pi\theta}(\frac{\pi}{2}-x)^2dx+\Re\int_0^{\pi\theta}\ln^2(1-e^{i2\pi x})dx-\pi\theta\ln^2(2)-2\ln(2)(-\pi\theta\ln2-\frac12\operatorname{Cl_2}(2\pi\theta))$$
Since $\displaystyle \ln^2(1-x)=2\sum_{n=1}^{\infty} \frac{H_{n-1}x^n}{n}$,
and using a closed form for $\sum_{n=1}^{\infty} \frac{H_n x^n}{n^2}$ obtained by @RaymondManzoni
We have $$=\frac{\pi^3}{12}\theta(4\theta^2-6\theta+3)+\pi\theta\ln^2(2)+\operatorname{Cl_2}(2\pi\theta)\ln2+2\sum_{n=1}^{\infty} \frac{H_{n-1}}{n}\int_0^{\pi\theta} \cos(2nx)dx
\\=\frac{\pi^3}{12}\theta(4\theta^2-6\theta+3)+\pi\theta\ln^2(2)+\operatorname{Cl_2}(2\pi\theta)\ln2+\Im[i\pi\theta\ln^2(1-e^{i2\pi\theta})]+\Im[\ln(1-e^{i2\pi x})\operatorname{Li_2}(1-e^{i2\pi\theta})]-\Im\operatorname{Li_3}(1-e^{i2\pi\theta})
$$ $$=\frac{\pi^3\theta^3}{3}-\pi\theta\ln^2(2\sin(\pi\theta))-\ln(\sin(\pi\theta)\operatorname{Cl_2}(2\pi\theta)+\pi\theta\ln^2(2)-\Im\operatorname{Li_3}(1-e^{i2\pi\theta}) \tag{3}$$
Putting $(2)$ and $(3)$ in $(1)$, a true miracle happens and the verbosity reduces down to
$$\int_0^{\pi\theta} \ln^2\left(\frac{\sin(x)}{\sin(\pi\theta)}\right)dx=\frac{\pi^3\theta^3}{3}-\Im\operatorname{Li_3}(1-e^{i2\pi\theta})$$,
Or in terms of your original function,
$$\mathcal{I}(a,b)=\frac{8}{\sqrt{b}}\left(\frac13\operatorname{arcsin}^3\sqrt{\frac{b}{a}}-\Im\operatorname{Li_3}\left(1-e^{2i\operatorname{arcsin}\sqrt{\tfrac{b}{a}}}\right)\right)$$
As a cute example, since $\sin(\pi/10)=\frac{\sqrt{5}-1}{4}$ we have
$$ \mathcal{I}(8,3-\sqrt{5})=\frac{\pi^3}{375\sqrt{3-\sqrt{5}}}-\frac{8}{\sqrt{3-\sqrt{5}}}\Im\operatorname{Li_3}(1-e^{i\pi/5})$$
Another one:
$$ \mathcal{I}(4,2-\sqrt{3})=\frac{\pi^3}{648\sqrt{2-\sqrt{3}}}-\frac{8}{\sqrt{2-\sqrt{3}}}\Im\operatorname{Li_3}\left(\frac{2-\sqrt{3}}{2}-\frac{i}{2}\right)$$
