I think that if $F,G$ are real symmetric positive semidefinite matrices, then $F^{\frac{1}{2}}GF^{\frac{1}{2}}$ is also symmetric positive semidefinite.
Consider the example by @user8675309, in which
$$
F = \begin{pmatrix}
13 & 43 \\
43 & 145
\end{pmatrix} \qquad
G = \begin{pmatrix}
117 & 87 \\
87 & 145
\end{pmatrix}
$$
Then
$$
F^{\frac{1}{2}} =
\frac{1}{\sqrt{170}}
\begin{pmatrix}
19 & 43 \\
43 & 151
\end{pmatrix} \qquad
G^{\frac{1}{2}} =
\frac{1}{\sqrt{257}}
\begin{pmatrix}
150 & 87 \\
87 & 74
\end{pmatrix}
$$
and
$$
F^{\frac{1}{2}}GF^{\frac{1}{2}} =
\frac{1}{17}
\begin{pmatrix}
45250 & 144754 \\
144754 & 465226
\end{pmatrix}
$$
The eigenvalues of $F^{\frac{1}{2}}GF^{\frac{1}{2}}$ are $\lambda_1 = 2(7507 - \sqrt{56270485})$ and $\lambda_2 = 2(7507 + \sqrt{56270485})$, which are both positive.
I found stated in this thread that $FG$ and $F^{\frac{1}{2}}GF^{\frac{1}{2}}$ have the same nonzero eigenvalues, and this example confirms it.
Let $P$ be an orthogonal matrix (a change from one orthogonal basis to another). Then $(P^{\top}F^{\frac{1}{2}}P)(P^{\top}GP)(P^{\top}F^{\frac{1}{2}}P) = P^{\top}(F^{\frac{1}{2}}GF^{\frac{1}{2}})P$. In other words the operation $(f, g) \mapsto h$ given by the following steps:
- pick an orthogonal basis $B$ of $V$;
- represent $f$ as a matrix $F$ w.r.t. $B$;
- represent $g$ as a matrix $G$ w.r.t. $B$;
- compute $H = F^{\frac{1}{2}}GF^{\frac{1}{2}}$;
- lef $h$ be the bilinear form associated to $H$ w.r.t. $B$.
is well-defined because it does not depend on the basis $B$. This is really the operation I cared about. Is it possible to define it without using the isomorphism between forms and matrices?