Find the value of $\alpha,\beta$ for the equation $\cos\alpha \cos\beta \cos(\alpha +\beta)=-\frac{1}{8}$
$\alpha>0$ & $\beta<\frac{\pi}{2}$
I get the following step after some substitution
$\cos2\alpha + \cos2\beta+\cos2(\alpha+\beta)=-\frac{3}{2}$ from here not able to proceed.
Suppose $$0<\alpha,\beta<\frac\pi2.\tag{1}$$ From $$\cos2\alpha + \cos2\beta+\cos2(\alpha+\beta)=-\frac{3}{2}$$ one has $$ \cos^2\alpha+\cos^2\beta+\cos^2(\alpha+\beta)=\frac34. $$ Note that $$\cos\alpha \cos\beta \cos(\alpha +\beta)=-\frac{1}{8}\tag{2}$$ implies $$\cos^2\alpha \cos^2\beta \cos^2(\alpha +\beta)=\frac{1}{64}.$$ By the AM-GM inequality $$ a+b+c\ge3\sqrt[3]{abc}$$ one has $$ \frac34=\cos^2\alpha+\cos^2\beta+\cos^2(\alpha+\beta)=3\sqrt[3]{\cos^2\alpha\cos^2\beta\cos^2(\alpha+\beta)}=3\sqrt[3]{\frac{1}{64}}=\frac34 $$ and the equal sign holds if and only if $$\cos^2\alpha=\cos^2\beta=\cos^2(\alpha+\beta).\tag{3}$$ From (1)(2)(3), it is easy to see $$ \alpha=\beta=\frac{\pi}{3}. $$
One has $$\cos \alpha\cos\beta(\cos\alpha\cos\beta - \sin\alpha\sin\beta) = -\frac{1}{8}$$ $$1 - \tan\alpha\tan\beta = -\frac{1}{8}(1+\tan^2\alpha)(1+\tan^2\beta)$$ $$8-8\tan\alpha\tan\beta = -1-\tan^2\alpha-\tan^2\beta - \tan^2\alpha\tan^2\beta$$ $$(\tan\alpha\tan\beta-3)^2 + (\tan\alpha-\tan\beta)^2 = 0$$ One then has $$\tan\alpha=\tan\beta=\sqrt{3}.$$
Like $ \cos {A} \cos {B} \cos {C} \leq \frac{1}{8} $,
$$-1=8\cos\alpha\cos\beta\cos(\alpha+\beta)=4[\cos(\alpha-\beta)+\cos(\alpha+\beta)]\cos(\alpha+\beta)$$
$$\iff\cos^2(\alpha+\beta)+\cos(\alpha+\beta)\cos(\alpha-\beta)+\dfrac14=0$$
which is a Quadratic Equation in $\cos(\alpha+\beta)$ which is real,
so, the discriminant must be $\ge0$
i.e., $$0\le\cos^2(\alpha-\beta)-1=-\sin^2(\alpha-\beta)$$
$$\implies(i)\sin(\alpha-\beta)=0$$
$\implies\alpha-\beta=m\pi$ where $m$ is any integer
As $0<\alpha,\beta<\dfrac\pi2,m=0\implies\cos(\alpha-\beta)=1$
and $$(ii)\cos(\alpha+\beta)=-\dfrac{\cos(\alpha-\beta)}2\implies\cos2\alpha=\dfrac12$$
