Solution without using beta function:
Let $a=\ln(\sin x)$ and $b=\ln(\cos x)$ in the algebraic identity
$$ab=\frac12a^2+\frac12b^2-\frac12(a-b)^2,$$
we have
$$\ln(\sin x)\ln(\cos x)=\frac12\ln^2(\sin x)+\frac12\ln^2(\cos x)-\frac12\ln^2(\tan x).$$
Integrate both sides from $x=0$ to $\pi/2$,
\begin{gather*}
\int_0^{\frac{\pi}{2}}\ln(\sin x)\ln(\cos x)\mathrm{d}x\\
=\frac12\int_0^{\frac{\pi}{2}}\ln^2(\sin x)\mathrm{d}x+\frac12\int_0^{\frac{\pi}{2}}\ln^2(\cos x)\mathrm{d}x-\frac12\int_0^{\frac{\pi}{2}}\ln^2(\tan x)\mathrm{d}x.
\end{gather*}
The second integral is equivalent to the first one by using the rule $\int_a^b f(x)\mathrm{d}x=\int_a^b f(a+b-x)\mathrm{d}x$. For the third integral, let $\tan x=y$, we have
$$\int_0^{\frac{\pi}{2}}\ln(\sin x)\ln(\cos x)\mathrm{d}x=\int_0^{\frac{\pi}{2}}\ln^2(\sin x)\mathrm{d}x-\frac12\int_0^\infty\frac{\ln^2(y)}{1+y^2}\mathrm{d}y.$$
First integral:
We have
$$\ln^2(2\sin x)=\ln^2(2)+2\ln(2)\ln(\sin x)+\ln^2(\sin x)$$
or
$$\ln^2(\sin x)=\ln^2(2\sin x)-2\ln(2)\ln(\sin x)-\ln^2(2).$$
Integrate both sides from $x=0$ to $\pi/2$,
$$\int_0^{\frac{\pi}{2}} \ln^2(\sin x)\mathrm{d}x=\int_0^{\frac{\pi}{2}}\ln^2(2\sin x)\mathrm{d}x-2\ln(2)\int_0^{\frac{\pi}{2}}\ln(\sin x)\mathrm{d}x-\int_0^{\frac{\pi}{2}}\ln^2(2)\mathrm{d}x.$$
The third integral is $\frac{\pi}{2}\ln^2(2)$ and the second integral is $-\frac{\pi}{2}\ln(2)$. For the first one, integrate both sides of the identity:
$$\ln^2(2\sin x)=\left(\frac{\pi}{2}-x\right)^2+2\sum_{n=1}^\infty\frac{H_{n-1}}{n}\cos(2nx)$$
from $x=0$ to $\pi/2$ then change the order of integration and summation,
\begin{gather*}
\int_0^{\frac{\pi}{2}}\ln^2(2\sin x)\mathrm{d}x=\int_0^{\frac{\pi}{2}}\left(\frac{\pi}{2}-x\right)^2\mathrm{d}x+2\sum_{n=1}^\infty\frac{H_{n-1}}{n}\int_0^{\frac{\pi}{2}}\cos(2nx)\mathrm{d}x\\
=-\frac13\left(\frac{\pi}{2}-x\right)^3\bigg|_0^{\frac{\pi}{2}}+2\sum_{n=1}^\infty\frac{H_{n-1}}{n}\cdot\frac{\sin(2nx)}{2n}\bigg|_0^{\frac{\pi}{2}}\\
=\frac{\pi^3}{24}+2\sum_{n=1}^\infty\frac{H_{n-1}}{n}\cdot\frac{\sin(n\pi)}{2n}\\
\{\text{the sum evaluates to $0$, since $\sin(n\pi)=0$ for integer $n$}\}\\
=\frac{\pi^3}{24}.
\end{gather*}
Therefore,
$$\int_0^{\frac{\pi}{2}} \ln^2(\sin x)\mathrm{d}x=\frac{\pi}{2}\ln^2(2)-\frac{\pi^3}{16}.$$
Second integral:
By the generalization:
$$ \int_0^\infty\frac{\ln^{2a}(x)}{1+x^2}\mathrm{d}x=2^{-2a-1}\pi\lim_{s\to \frac12}\frac{\mathrm{d}^{2a}}{\mathrm{d} s^{2a}}\csc(\pi s),$$
which follows from differentiating Euler's reflection formula, we have
$$ \int_0^\infty\frac{\ln^2(x)}{1+x^2}\mathrm{d}x=\frac{\pi^3}{8}.$$
Combining the two integrals, we reach
$$\int_0^{\frac{\pi}{2}} \ln(\sin x)\ln(\cos x)\mathrm{d}x=\frac{\pi}{2}\ln^2(2)-\frac{\pi^3}{48}.$$