Let $0\lt x\le y\le z\lt\pi$ be the three half-angles of the triangle, with $x+y+z=\pi$. Then with the comment by @Gribouillis the condition $ac\gt b^2$ translates to
$$
\sin x\sin z\gt\sin^2y
$$
and thus, with $\sin z=\sin(\pi-x-y)=\sin(x+y)$, to
$$
\sin x\sin(x+y)\gt\sin^2 y\;.
$$
Applying the addition theorem yields
$$
\sin x(\sin x\cos y+\cos x\sin y)\gt\sin^2y\;.
$$
To solve for $x$, isolate $\cos x$,
$$
\cos x\gt\frac{\sin y}{\sin x}-\frac{\sin x}{\sin y}\cos y\;,
$$
square,
$$
\cos^2 x\gt\frac{\sin^2 y}{\sin^2 x}+\frac{\sin^2 x}{\sin^2 y}\cos^2y-2\cos y\;,
$$
use $\sin^2x+\cos^2x=1$ and $\sin^2y+\cos^2y=1$ and simplify:
$$
1+2\cos y\gt\frac{\sin^2 y}{\sin^2 x}+\frac{\sin^2 x}{\sin^2 y}\;.
$$
This is a quadratic inequality for $\frac{\sin^2 x}{\sin^2 y}$, and the relevant solution branch in our case is
$$
\frac{\sin^2 x}{\sin^2 y}\gt\frac12+\cos y-\sqrt{\left(\frac12+\cos y\right)^2-1}\;.
$$
Thus the bound for $x$ as a function of $y$ is
$$
x\gt f(y):=\arcsin\left(\sin y\sqrt{\frac12+\cos y-\sqrt{\left(\frac12+\cos y\right)^2-1}}\right)\;.
$$
The range for $x$ is $0\lt x\le y$ if $y\le\frac\pi3$ and $0\lt x\le1-2y$ if $\frac\pi3\le y\lt\frac\pi2$. In the latter case, we never have $ac\gt b^2$, so the desired probability is
\begin{eqnarray*}
1-\frac{\int_0^\frac\pi3f(y)\mathrm dy+\int_\frac\pi3^\frac\pi2(1-2y)\mathrm dy}{\int_0^\frac\pi3y\mathrm dy+\int_\frac\pi3^\frac\pi2(1-2y)\mathrm dy}
&=&
1-\frac{\int_0^\frac\pi3f(y)\mathrm dy+\frac{\pi^2}{36}}{\frac{\pi^2}{18}+\frac{\pi^2}{36}}
\\
&=&\frac23-\frac{12}{\pi^2}\int_0^\frac\pi3f(y)\mathrm dy\;.
\end{eqnarray*}
This is $\frac15$ if we have
\begin{eqnarray*}
\int_0^\frac\pi3f(y)\mathrm dy
&=&
\frac7{180}\pi^2
\\
&\approx&
0.383817948931253\;,
\end{eqnarray*}
and indeed this is what a numerical Wolfram|Alpha integration to $15$ decimal places yields.
I haven’t been able to solve the integral analytically, but in any case, this is a very inelegant approach – a simple result like $\frac15$ for a geometric probability usually has a nice explanation based on symmetry. I was hoping that perhaps one could show that exactly $2$ of the $10$ triangles formed by $5$ points fulfil the condition, but this is far from the case; that number varies between $0$ and $5$. Still, I wouldn’t be surprised if someone comes up with an elegant solution.
$$ \frac13\left(0+\frac15+1\right)=\frac25 $$
in the unordered case. That might be a bit easier to deal with.
– joriki Jan 02 '24 at 19:53