Thanks for all the commentators. I solved this problem with the help of relevant links, but slightly different from them.
First, integrate by parts
$\begin{align} I&=\int_{0}^{\frac{\pi}{2}}\frac{x\ln{(\cos x)}}{\tan x}dx=\int_{0}^{\frac{\pi}{2}}x\ln{(\cos x)}d(\ln(\sin x))\\ &=x\ln{(\cos x)}\ln(\sin x)\left|_{0}^{\pi/2}\right.-\int_{0}^{\frac{\pi}{2}}\ln{(\sin x)}d(x\ln{(\cos x)})dx\\ &=-\int_{0}^{\frac{\pi}{2}}\ln{(\sin x)}(\ln{(\cos x)}-x\tan x)dx\\ &=-\int_{0}^{\frac{\pi}{2}}\ln{(\sin x)}\ln{(\cos x)}dx+\frac{\pi}{2}\int_{0}^{\frac{\pi}{2}} \tan x\ln{(\sin x)}dx-\int_{0}^{\frac{\pi}{2}}(\frac{\pi}{2}-x) \tan x\ln{(\sin x)}dx\\ &=-\int_{0}^{\frac{\pi}{2}}\ln{(\sin x)}\ln{(\cos x)}dx+\frac{\pi}{2}\int_{0}^{\frac{\pi}{2}} \tan x\ln{(\sin x)}dx-I \end{align}$
then these two integrals can be solved as follows
$\displaystyle I_1=\int_{0}^{\frac{\pi}{2}}\ln{(\sin x)}\ln{(\cos x)}dx=\frac{\pi}{2} \ln ^{2}(2)-\frac{\pi^{3}}{48}\ $ (link)
$\displaystyle I_2=\int_{0}^{\frac{\pi}{2}} \tan x\ln{(\sin x)}dx=-\frac{\pi^2}{24}\ $(link)
therefore $\ I=\dfrac{-I_1+\dfrac{\pi}{2}I_2}{2}=-\dfrac{\pi}{4}\ln^2 (2)$