I have run a FORTRAN code and I have obtained strong evidence that $$\int_0^{\pi/3} \!\! \big((\sqrt{3}\cos\vartheta-\sin\vartheta)\sin\vartheta\big)^{\!1/2}\!\cos\vartheta \,d\vartheta =\frac{\pi\sqrt{3}}{8\sqrt{2}}. $$ In fact, it looks like my numerical integration method (the trapezoid rule) converges to the value $\dfrac{\pi\sqrt{3}}{8\sqrt{2}}$, with at least $12$ significant digits.
Any ideas how to prove this result analytically?
First, an identity:
$$
\begin{align}
(\sqrt{3}\cos(\theta)-\sin(\theta))\sin(\theta)
&=2\sin(\tfrac\pi3-\theta)\sin(\theta)\\
&=\cos(\tfrac\pi3-2\theta)-\cos(\tfrac\pi3)\\
&=\tfrac12-2\sin^2(\tfrac\pi6-\theta)\\
&=\tfrac12-2\sin^2(\theta-\tfrac\pi6)\tag{1}
\end{align}
$$
Now, the integral
$$
\begin{align}
&\int_0^{\pi/3}
\left((\sqrt{3}\cos(\theta)-\sin(\theta))\sin(\theta)\right)^{1/2}\cos(\theta)\,\mathrm{d}\theta\tag{2}\\
&=\int_0^{\pi/3}
\left(\tfrac12-2\sin^2(\theta-\tfrac\pi6)\right)^{1/2}\cos(\theta)\,\mathrm{d}\theta\tag{3}\\
&=\frac1{\sqrt2}\int_{-\pi/6}^{\pi/6}
\left(1-4\sin^2(\theta)\right)^{1/2}\cos(\theta+\tfrac\pi6)\,\mathrm{d}\theta\tag{4}\\
&=\frac1{\sqrt2}\int_{-\pi/6}^{\pi/6}
\left(1-4\sin^2(\theta)\right)^{1/2}\left(\tfrac{\sqrt3}{2}\cos(\theta)-\tfrac12\sin(\theta)\right)\,\mathrm{d}\theta\tag{5}\\
&=\frac{\sqrt3}{2\sqrt2}\int_{-\pi/6}^{\pi/6}
\left(1-4\sin^2(\theta)\right)^{1/2}\cos(\theta)\,\mathrm{d}\theta\tag{6}\\
&=\frac{\sqrt3}{2\sqrt2}\int_{-\pi/6}^{\pi/6}
\left(1-4\sin^2(\theta)\right)^{1/2}\,\mathrm{d}\sin(\theta)\tag{7}\\
&=\frac{\sqrt3}{4\sqrt2}\int_{-\pi/2}^{\pi/2}
\left(1-\sin^2(\theta)\right)^{1/2}\,\mathrm{d}\sin(\theta)\tag{8}\\
&=\frac{\sqrt3}{4\sqrt2}\int_{-\pi/2}^{\pi/2}
\cos^2(\theta)\,\mathrm{d}\theta\tag{9}\\
&=\frac{\pi\sqrt3}{8\sqrt2}\tag{10}
\end{align}
$$
Explanation:
$\ \:(3)$: Apply $(1)$
$\ \:(4)$: subsitute $\theta\mapsto\theta+\frac\pi6$
$\ \:(5)$: cosine of a sum formula
$\ \:(6)$: toss the odd part over a symmetric domain
$\ \:(7)$: $\mathrm{d}\sin(\theta)=\cos(\theta)\,\mathrm{d}\theta$
$\ \:(8)$: subsitute $\sin(\theta)\mapsto\frac12\sin(\theta)$
$\ \:(9)$: $\sqrt{1-\sin^2(\theta)}=\cos(\theta)$ and $\mathrm{d}\sin(\theta)=\cos(\theta)\,\mathrm{d}\theta$
$(10)$: $\int_{-\pi/2}^{\pi/2}\cos^2(\theta)\,\mathrm{d}\theta=\frac\pi2$
OK, you can start by noting that
$$\sqrt{3} \cos{x} - \sin{x} = 2 \sin{\left ( \frac{\pi}{3}-x\right)}$$
which means that the integral is
$$ \sqrt{2} \int_0^{\pi/3} dx \, \cos{x} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}} = \sqrt{2} I$$
You can make a substitution $x \mapsto \frac{\pi}{3}-x$ and see that
$$I = \sqrt{3} \int_0^{\pi/3} dx \, \sin{x} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}}$$
EDIT
Integrate by parts:
$$\begin{align}I &= \left [\sin{x} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}} \right ]_0^{\pi/3} - \int_0^{\pi/3} dx \, \sin{x} \frac{d}{dx} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}}\\ &= -\frac12 \int_0^{\pi/3} dx \, \sin{x} \frac{\cos{x} \sin{\left ( \frac{\pi}{3}-x\right)} - \sin{x} \cos{\left ( \frac{\pi}{3}-x\right)}}{\sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}}}\\ &= -\frac12 I + \frac12 \int_0^{\pi/3} dx \, \sin{x} \sqrt{\frac{\sin{x}}{\sin{\left ( \frac{\pi}{3}-x\right)}} } \cos{\left ( \frac{\pi}{3}-x\right)} \end{align}$$
This means that
$$\begin{align}3 I &= \int_0^{\pi/3} dx \, \frac{\cos{x}}{\sin{x}} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}} \cos{\left ( \frac{\pi}{3}-x\right)}\\ &= \frac{\sqrt{3}}{2} \int_0^{\pi/3} dx \, \frac{\cos^2{x}}{\sin{x}} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}} - \frac12 I \end{align}$$
Combining again...
$$7 I = \sqrt{3} \int_0^{\pi/3} dx \sqrt{\frac{\sin{\left ( \frac{\pi}{3}-x\right)}}{\sin{x}}} - \underbrace{\sqrt{3} \int_0^{\pi/3} dx \, \sin{x} \sqrt{\sin{x} \sin{\left ( \frac{\pi}{3}-x\right)}}}_{\text{We know from above this equals } I}$$
Thus
END EDIT
$$I = \frac{\sqrt{3}}{8} \int_0^{\pi/3} dx \sqrt{\frac{\sin{x}}{\sin{\left ( \frac{\pi}{3}-x\right)}}}$$
Now, in a similar manipulation as in the evaluation of this integral, sub $u = \sin{x}/\sin{(\pi/3-x)}$ and find that the integral becomes
$$I = \frac{3}{16} \int_0^{\infty} du \frac{\sqrt{u}}{1+u+u^2}$$
This integral is very straightforward to evaluate via residues using, e.g., a keyhole contour about the positive real axis. By the residue theorem, the original integral is then
$$\sqrt{2} I = \frac{3 \sqrt{2}}{16} \frac12 i 2 \pi \left (\frac{e^{i \pi/3}}{i \sqrt{3}} - \frac{e^{i 2 \pi/3}}{i \sqrt{3}} \right ) = \frac{\pi}{8} \sqrt{\frac{3}{2}}$$
