ADDED: Another way of saying this: Given integers $B > A > 0,$ with $\gcd(A,B) = 1,$ then there is a solution in integers $x,y,z,$ not all $0,$ to
$$ A(x^2 + y^2 + z^2) - B (yz + zx + xy) = 0, $$
if and only if both $B+2A$ and $B-A$ are integrally represented by the binary form $u^2 + 3 v^2.$
Well, as i indicated, we can multiply the form by $12$ and diagonalize that over the integers. That will not always be the best way to proceed, but gives an immediate answer in one case the computer program suggested.
Given integers $B > A > 0,$ with $\gcd(A,B) = 1,$ let
$$ g(x,y,z) = A(x^2 + y^2 + z^2 ) - B(yz+zx+xy). $$
Before continuing, let me point out that the discriminant of $g$ is
$$ \Delta = -(B-A) (2A+B)^2, $$ and that primes that can be used to rule out solutions of $g=0$ are required to divide $\Delta.$ Furthermore, as the form is indefinite, the number of primes for which we can make such an argument is even: for $A=3,B=14$ we had $ \Delta = -11 \cdot 20^2,$ and the primes that can be used to prove that one impossible are $5,11.$
Then, defining
$$ u = -x-y+2z, \; \; \; v = -x+y, \; \; \; w = x+y+z, $$ we get diagonalization
$$ 12 g = (2A+B) u^2 + 3 (2A+B) v^2 -4(B-A) w^2. $$
Now, suppose $B$ is divisible by $4.$ We can prove, quickly, that it is impossible to have the above $12g = 0$ in integers, so we cannot have $g=0$ either. IF $B=4 \beta,$ we know that $A$ is odd to have them coprime. So,
$$ 12g = (2A+4 \beta) u^2 + 3 (2A + 4 \beta)v^2 - 4 (4 \beta - A) w^2 $$
is to be set to zero. However, all coefficients are currently even, so we immediately switch to
$$ 6g = (A+2 \beta) u^2 + 3 (A + 2 \beta)v^2 - 2 (4 \beta - A) w^2. $$
Now, both $A+2\beta$ and $4 \beta - A$ are odd. Using properties of $p^2 + 3 q^2,$ we find that the quantity we abbreviate as $6g$ cannot be divisible by $8$ unless all three of $u,v,w$ are even. Therefore there is no solution in coprime integers to $6g=0,$ and by infinite descent, there is no solution in integers. There is a lemma here, if there is any integer solution, we can divide through by the gcd of the variables, and arrive at a solution with gcd $1.$ But that does not happen; so there cannot be a solution in integers to $g(x,y,z) = 0.$ There is a phrase for this: $g(x,y,z)$ is anisotropic in the $2$-adic numbers $\bf Q_2.$
So, $B$ can't be divisible by $4.$
Same argument if $A+B$ is divisible by $4.$ From $\gcd(A,B) = 1,$ this means both are odd. Then, with $B > A > 0,$ we find $B-a \equiv 2 \pmod 4.$ So, with
$$ 12 g = (2A+B) u^2 + 3 (2A+B) v^2 -4(B-A) w^2, $$
we also get $2A+B$ odd. Let us write $$ 2A + B=\psi , \; \; \; B_A = 2 \omega $$
with $\psi, \omega$ odd. We have
$$ 12 g = \psi( u^2 + 3v^2) - 8 \omega \, w^2. $$
This is not divisible by $16$ unless all three $u,v,w$ are even. So this is again anisotropic in $\bf Q_2.$
So, we cannot have $A+B$ divisible by $4.$
Next, $2A+B$ and $B-A$ cannot both be divisible by $2$ to an odd power if the original form is isotropic. If $B$ is odd, then $2A+B$ is odd, and if $\nu_2 (B-A)$ is odd, the the original for is anisotropic in $\bf Q_2.$ Here $\nu_2$ is the $2$-adic valuation, if we say $\nu_2 n = k$ it means $2^k | n$ but NOT $2^{k+1} | n.$ Next, if $B$ is even we know that $B \neq 0 \pmod 4$ if the thing is isotropic, so actually $B \equiv 2 \pmod 4,$ and then $A$ is odd. But then $B-A$ is odd, so $\nu_2 4(B-A) = 2$ is even. If the original form is isotropic, we must have $\nu_2 (2A+B)$ even as well.
% Feb. 6
From $\gcd(A,B) = 1.$ Then, with $B > A > 0.$
$$ 12 g = (2A+B) u^2 + 3 (2A+B) v^2 -4(B-A) w^2. $$
Note that
$$ \gcd(2A+B, 3(2A+B), -4(B-A) ) \in \{1,2,3,4,6,12 \}. $$
In particular, we can say something pretty strong when either is divisible by some prime
$$ q \geq 5, \; \; \; q \equiv 2 \pmod 3. $$
Theorem 1: if $2A+B$ is divisible by such a prime $q$ but not by $q^2,$ then there is no solution in integers to our $12g = 0,$ because the form is anisotropic in $\bf Q_q.$
Theorem 2: if $B-A$ is divisible by such a prime $q$ but not by $q^2,$ then there is no solution in integers to our $12g = 0,$ because the form is anisotropic in $\bf Q_q.$
Proofs: if $$ u^2 + 3 v^2 \equiv 0 \pmod q, $$ then actually
$$ u,v \equiv 0 \pmod q, $$
$$ u^2 + 3 v^2 \equiv 0 \pmod {q^2}. $$
So, for example, this immediately takes care of $A=3, B=14$ because $B-A = 11$ and $2B+A = 4 \cdot 5.$
The same conclusions hold if either quantity named is divisible by $q^{2n-1}$ but not by $q^{2n}.$
The primes in question are
$$ 5, 11, 17, 23, 29, 41, 47, 53, 59, 71, 83, 89, 101, 107, \ldots $$
If either $2A+B$ or $B-A$ is divisible by one of these primes, but not its square, we know there cannot be a way to write zero in integers for that pair $(A,B).$ I am running a program showing $(A,B)$ pairs that fail, then showing the other two numbers, $2A+B$ and $B-A$ and how each factors. In most cases of failure one can easily find one or two of these primes present.
% Feb. 14
It is simpler than I had feared. We take $0 < A < B,$ and $\gcd(A,B)=1.$ After that, what we are really concerned about are the two numbers that come up in diagonalizing the form, those being $B + 2A$ and $B-A.$
The form is isotropic over the rationals (and integers) if and only if:
(I) when factoring both $B + 2A$ and $B-A,$ the exponents of $2$ are even.
(II) when factoring both $B + 2A$ and $B-A,$ the exponents of $q$ are even, where $q \equiv 5 \pmod 6$ is a prime.
That's it. Note that we could combine these into one test, for all primes $p \equiv 2 \pmod 3.$
Another way of saying this: Given integers $B > A > 0,$ with $\gcd(A,B) = 1,$ then there is a solution in integers $x,y,z,$ not all $0,$ to
$$ A(x^2 + y^2 + z^2) - B (yz + zx + xy) = 0, $$
if and only if both $B+2A$ and $B-A$ are integrally represented by the binary form $u^2 + 3 v^2.$
% Feb. 16 2015
Turned out finding $x,y,z$ can be made mechanical as well, once $2A+B$ and $B-A$ are represented by $u^2 + 3 v^2.$ Notice that I actually use $4(B-A)$ instead!
Given
$$ p^2 + 3 q^2 = 2 A + B, $$
$$ r^2 + 3 s^2 = 4(B-A), $$
we can solve
$$ A(x^2 + y^2 + z^2) = B (yz + zx + xy) $$
with
$$ x = 2 p^2 + 6 q^2 - p r - 3 p s + 3 q r - 3 q s, $$
$$ y = 2 p^2 + 6 q^2 - p r + 3 p s - 3 q r - 3 q s, $$
$$ z = 2 p^2 + 6 q^2 + 2 p r + 6 q s. $$
I took $g = \gcd(x,y,z)$ and printed $x/g,y/g,z/g,$ it changes nothing.
A B 2A+B 4(B-A) p q r s x y z
1 2 4 4 2 0 2 0 1 1 4 18=18
2: 1
2 3 7 4 2 1 2 0 8 2 11 378=378
3: 1
A B 2A+B 4(B-A) p q r s x y z
4: 0
1 5 7 16 2 1 4 0 3 -1 5 35=35
2 5 9 12 3 0 3 1 0 1 2 10=10
4 5 13 4 1 2 2 0 6 2 5 260=260
5: 3
5 6 16 4 4 0 2 0 1 1 2 30=30
6: 1
3 7 13 16 1 2 4 0 23 -1 17 2457=2457
6 7 19 4 4 1 2 0 6 4 9 798=798
7: 2
% Feb. 18, 2015
Given integers $B > A > 0,$ with $\gcd(A,B) = 1,$ let
$$ g(x,y,z) = A(x^2 + y^2 + z^2 ) - B(yz+zx+xy). $$
Then, defining
$$ r = x+y+z, \; \; \; s = x-y, \; \; \; t = y-z, $$ we get almost diagonalization
$$ 3 g = (A-B) r^2 + (2A+B) (s^2 + st + t^2 ) . $$
For me, this makes the prime $2$ easier to see, in that $s^2 + st + t^2$ is even if and only if both $s,t$ are even, in which case the $s^2 + st + t^2$ is divisible by $4.$ Can we have $g$ isotropic in $\bf Z$ and $B$ divisible by $4?$ No, because $\gcd(A,B) = 1,$ so $A$ is odd, so $2A+B \equiv 2 \pmod 4$ while $A-B$ is odd, so the thing is anisotropic in $\bf Q_2:$ we would need $r$ even, but then $s^2+st+t^2$ must be even and so $s,t$ are even as well. Can we have $g$ isotropic in $\bf Z$ and $A+B$ divisible by $4?$ No, because $\gcd(A,B) = 1,$ so both $A,B$ are odd, so $A-B \equiv 2 \pmod 4$ while $2A+B$ is odd, so the thing is anisotropic in $\bf Q_2:$ we would need $s^2+st+t^2$ even so $s,t$ are even, but then $r^2$ is even and $r$ is even as well. Can we have $g$ isotropic and both $\nu_2 \, (A-B)$ odd and $\nu_2 \, (2A+B)$ odd? No, this means $B$ even, but then $A$ is odd, and $(A-B)$ is odd.
In brief, in order to have $g$ isotropic in $\bf Z,$ we need both $\nu_2 \, (A-B)$ and $\nu_2 \, (2A+B)$ even.
Put this with the same observation about primes $q \equiv 5 \pmod 6,$ and it follows that if $g$ isotropic in $\bf Z,$ then both $ |A-B|$ and $ (2A+B)$ are represented by $s^2 + st + t^2$ or, indeed, $u^2 + 3 v^2.$
The way this worked out, I do not think a student needs to know much of anything about quadratic forms in more than two variables, if these lemmas about binary quadratic forms can be accepted without proof; the first one is easy and breifly discussed above. I think I will revert to letters $x,y.$ I'm afraid the proof of Lemma 2 uses one case of quadratic reciprocity, not much to be done about that. Lemma 4 uses a variety of ingredients; the hardest case is primes that are $1 \pmod 3,$ after that it is just mathematical induction on the number of prime factors dividing the number I'm calling $n.$
Define $\nu_p \, n$ to be the highest exponent of the prime $p$ that gives a divisor of $n.$ That is, $\nu_p \, n = k$ means
$p^k | n$ but NOT $p^{k+1} | n.$
Lemma 1: if $x^2 + xy + y^2$ is even, then both $x,y$ are even and $x^2 + xy + y^2$ is divisble by $4.$ Therefore we can divide both $x,y$ by $2,$ and repeat. The result is, $\nu_2 \, (x^2 + x y + y^2)$ is even.
Lemma 2: Given a prime $q \equiv 5 \pmod 6:$ if $x^2 + xy + y^2$ is divisible by $q,$ then both $x,y$ are divisible by $q,$ and $x^2 + xy + y^2$ is divisble by $q^2.$ Therefore we can divide both $x,y$ by $q,$ and repeat. The result is, $\nu_q \, (x^2 + x y + y^2)$ is even.
Lemma 3: $x^2 + xy + y^2$ and $x^2 + 3 y^2$ represent exactly the same numbers
Lemma 4: If we have a positive number $n$ such that, for every prime $p \equiv 2 \pmod 3$ we have $\nu_p \, n$ even, then we can find some integers $x,y$ such that $n = x^2 + 3 y^2.$ We can also find other integers such that $n = x^2 + xy + y^2,$ but these $x,y$ may sometimes have opposite $\pm$ signs.
