This problem is for me more a "recreational" one, but might even have no solution or better: is part of an open problem. The set of equations comes from the problem of "magic 3x3 square of squares" where the $9$ unknowns are written as $$M = \small \begin{bmatrix} a^2 & b^2 & c^2 \\ d^2 & e^2 & f^2 \\ g^2 & h^2 & i^2 \\ \end{bmatrix} $$ where row, column and diagonal sums shall equal the same value (say $3 e^2$). By the rules of magic squares all values $a²,b²,...,i²>0$ shall be pairwise distinct.
I managed to get the circular sets of linear equations in quadratics $$\tag 1 \small \begin{bmatrix} -1 & 1 & 1 \\ 4 & -1 & -2 \\ 2 & 0 & -1 \\ \end{bmatrix} \cdot \small \begin{bmatrix} e^2 \\ h^2 \\ i^2 \\ \end{bmatrix} = \small \begin{bmatrix} c^2 \\ f^2 \\ a^2 \\ \end{bmatrix} $$
$$\tag 2 \small \begin{bmatrix} 2 & 3 & -4 \\ 1 & 2 & -2 \\ 2 & 1 & -2 \\ \end{bmatrix} \cdot \small \begin{bmatrix} c^2 \\ f^2 \\ a^2 \\ \end{bmatrix} = \small \begin{bmatrix} b^2 \\ d^2 \\ g^2 \\ \end{bmatrix} $$
$$\tag 3 \small \begin{bmatrix} -1/2 & 1 & 1/2 \\ -2 & 2 & 1 \\ 1/2 & 0 & 1/2 \\ \end{bmatrix} \cdot \small \begin{bmatrix} b^2 \\ d^2 \\ g^2 \\ \end{bmatrix} = \small \begin{bmatrix} e^2 \\ h^2 \\ i^2 \\ \end{bmatrix} $$ $ \qquad \qquad $ (Matrix (2) has been corrected against first version due to the comment)
I think that this is an interesting diophantine problem in itself: to handle connected equations in $4$ squares. By L. Euler we know already parametrizations for possible solutions for $4$-squares-equations, but I'm not familiar enough with that to try to apply this here an to get -possibly- contradictions or an infinite ascent (or a solution). Likewise we have at least two connected Pell-equations (where some zero matrix-coefficients occur). I know already that all squares must be congruent $1$ to modulus $24$ (the unsquared values $a,b,c...,i= \pm1 \pmod 6$) and we have at most three free parameters (I start with $e^2,h^2,i^2$ as independents).
For instance, we can extract a set of connected equalities, each only involving three variables, which are very similar to Pell-equations.
$$\small \begin{array} {} 2i^2 &= b^2+d^2 \\ 2g^2 &= b^2+f^2 \\ 2e^2 &= a^2+i^2 &= b^2+h^2 &= c^2+g^2 &= d^2+f^2 \\ 2c^2 &= d^2+h^2 \\ 2a^2 &= f^2+h^2 \\ \end{array}$$
Q1: How could I possibly improve that ansatz?
or even:
Q2: Is a nonzero solution (besides a trivial solution with all $a=b=...=i=1$ which is excluded by definition of the problem) possible?
Looking at what I've observed so far
1) Hmm. I do not really look farther with this, but when I write the system of equations in a joint structure of a matrix-diagonalization, I get first this:
$$
M= \small \begin{bmatrix}
0 & 0 & 0 & 0 & 0 & 0 & -1 & 1 & 1 \\
0 & 0 & 0 & 0 & 0 & 0 & 4 & -1 & -2 \\
0 & 0 & 0 & 0 & 0 & 0 & 2 & 0 & -1 \\
2 & 3 & -4 & 0 & 0 & 0 & 0 & 0 & 0 \\
1 & 2 & -2 & 0 & 0 & 0 & 0 & 0 & 0 \\
2 & 1 & -2 & 0 & 0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & -1/2 & 1 & 1/2 & 0 & 0 & 0 \\
0 & 0 & 0 & -2 & 2 & 1 & 0 & 0 & 0 \\
0 & 0 & 0 & 1/2 & 0 & 1/2 & 0 & 0 & 0
\end{bmatrix}\cdot
\small \begin{bmatrix}
c^2\\f^2\\a^2\\b^2\\d^2\\g^2\\e^2\\h^2\\i^2
\end{bmatrix} =
\small \begin{bmatrix}
c^2\\f^2\\a^2\\b^2\\d^2\\g^2\\e^2\\h^2\\i^2
\end{bmatrix}
$$
By this $M$ the vector of the squares of the variables $c^2,f^2,...i^2$ is an eigenvector to the eigenvalue $1$; the eigenvalues are the three cube-roots of complex unit and each occurs threefold. We have only three real eigenvectors and they just encode the composition of $c^2,f^2,...i^2$ by the ("independents") $e^2,h^2,i^2$.
...
2) The third power $M^3$ is just the identity matrix, so no new insights come from here.
3) ... here my "expertise" in linear algebra is at its limit at the moment and possibly a helpful hint could emerge from that observations/speculations so far.