We will prove that
$$I=-\frac{\pi^4}{2880}.$$
Indeed, let
$$
J=\int_0^{\pi/2}\log^2(\sin x)\log(\cos x)\tan x \,dx
$$
It is easy to see that
$$\eqalign{J&=\int_0^{\pi/4}\log^2(\sin x)\log(\cos x)\tan x \,dx+
\int_{\pi/4}^{\pi/2}\log^2(\sin x)\log(\cos x)\tan x \,dx\cr
&=\int_0^{\pi/4}\log^2(\sin x)\log(\cos x)\tan x \,dx+
\int_{0}^{\pi/4}\log^2(\cos x)\log(\sin x)\cot x \,dx\cr
&=I
}$$
Now, to calculate $J$ we make the substitution $t\leftarrow\sin^2x$:
$$
J=\frac{1}{16}\int_0^1\frac{\log(1-u)}{1-u}\log^2(u)\,du
$$
But
$$\frac{\log(1-u)}{1-u}=-\left(\sum_{n=0}^\infty u^n\right)\left(\sum_{n=1}^\infty \frac{u^n}{n}\right)
=-\sum_{n=1}^\infty H_nu^n
$$
where $H_n=\sum_{k=1}^n1/k$. Hence
$$J=-\frac{1}{16}\sum_{n=1}^\infty H_n\int_0^1u^n\log^2(u)du
=-\frac{1}{8}\sum_{n=1}^\infty\frac{ H_n}{(n+1)^3} $$
Finally, since $H_{n}=H_{n+1}-\frac{1}{n+1}$, we get
$$J=\frac{1}{8}\zeta(4)-\frac{1}{8}\sum_{n=1}^\infty\frac{H_n}{n^3}\tag{1}$$
The sum $\sum_{n=1}^\infty\frac{H_n}{n^3}$ is known, it can be evaluated as follows, first we have
$$
H_n=\sum_{k=1}^\infty\left(\frac{1}{k}-\frac{1}{k+n}\right)=
\sum_{k=1}^\infty \frac{n}{k(k+n)}
$$
Thus
$$
\sum_{n=1}^\infty\frac{H_n}{n^3}=\sum_{k,n\geq1}\frac{1}{n^2k(n+k)}
=\sum_{k,n\geq1}\frac{1}{k^2n(n+k)}
$$
Taking the half sum we find
$$
\sum_{n=1}^\infty\frac{H_n}{n^3}=\frac{1}{2}\sum_{k,n\geq1}\frac{1}{kn(k+n)}\left(\frac{1}{k}+\frac{1}{n}\right)=
\frac{1}{2}\sum_{k,n\geq1}\frac{1}{k^2n^2}=\frac{1}{2}\zeta^2(2)
$$
Replacing in $(1)$ we obtain
$$J=\frac{1}{8}\zeta(4)-\frac{1}{16}\zeta^2(2)=-\frac{\pi^4}{2880}.$$
as announced.$\qquad\square$