The general solution to such puzzle is to write a general form of the function of $A$ and $B$ that the problem statement allows, and solve using the givens. So for example if we posit $f(A,B)=u\,A^2+v\,A\,B+w\,B^2+x\,A+y\,B+z$, we know $f(A,B)$ for 8 pairs of inputs, and that gives us 8 linear equations of the 6 unknowns $u$, $v$, $w$, $x$, $y$, $z$. If we find a solution to that, we have solved the puzzle.
Here the system goes:
$$\begin{array}{r}
u&+&25\;v&+&625\;w&+&x&+&25\;y&+&z&=&30\\
4\;u&+&100\;v&+&2500\;w&+&2\;x&+&50\;y&+&z&=&104\\
9\;u&+&210\;v&+&4900\;w&+&3\;x&+&70\;y&+&z&=&213\\
16\;u&+&232\;v&+&3364\;w&+&4\;x&+&58\;y&+&z&=&234\\
25\;u&+&160\;v&+&1024\;w&+&5\;x&+&32\;y&+&z&=&161\\
36\;u&+&594\;v&+&9801\;w&+&6\;x&+&99\;y&+&z&=&594\\
49\;u&+&147\;v&+&441\;w&+&7\;x&+&21\;y&+&z&=&146\\
64\;u&+&560\;v&+&4900\;w&+&8\;x&+&70\;y&+&z&=&558
\end{array}$$
and has (single) solution
$$u=0,v=1,w=0,x=-1,y=0,z=6$$
Thus $f(A,B)=A\,B-A+6$, thus $f(9,66)=591$.
It's a simple matter with the right tools. Try it online!.
Here $f$ is so simple that it was probably possible to find the solution by intuition, trial and error. But why bother when there is math?
More seriously: this shows that using any second-degree polynomial of $A$ and $B$ does not make the problem hard. This generalizes to any degree and in any field, including finite, e.g. the integers modulo $p$. This has applications in crypto, e.g. Shamir secret sharing can be thought as a variant of that where we reconstruct the coefficients of a polynomial given enough examples.