Given $a,b\in \mathbb N$, can $a^2+a b+b^2$ and $a^2- a b+b^2$ both be square numbers?

Question

The question arises if you search for two integer triangles ABC and AB'C with common vertices A and C, CA=b, CB=a, CB'=a+b and $\measuredangle ACB$ = $\measuredangle ACB'= 60°$. With c=AB and c'=AB' you get $c^2=a^2- a b+b^2$ and $c'^2=a^2+ a b+b^2$. I cannot find any $a,b\in \mathbb N$ such that $c,c'\in \mathbb N$. Is there an argument that shows that this is impossible?

Answer 1

There are no positive integer solutions to these equations. One way of proving this is to consider the Diophantine equation

$$a^4+a^2b^2+b^4=(a^2+ab+b^2)(a^2-ab+b^2)=z^2.$$ We can tackle this by the method used in Does the equation $y^2=3x^4-3x^2+1$ have an elementary solution?

Theorem

There are no positive integer solutions of either of the equations $$ x^4+x^2y^2+y^4=z^2\tag{1}$$ $$x^4-2x^2y^2-3y^4=z^2\tag{2}.$$

Proof

First note that, for either equation, we can suppose that the variables are pairwise coprime since a common factor of any pair of variables would be a factor of all and cancellation can occur.

An equation of form (1)

We can suppose that $x$ is odd. Then $z$ is odd and we have $y^2+y^4\equiv 0 \pmod 8$. Therefore $y$ is divisible by $4$ and we can write $y=2t$, where $t$ is even.

$ x^4+4x^2t^2+16t^4=z^2$ can be rewritten, using completing the square, as $$\left (\frac{x^2+2t^2-z}{2}\right )\left (\frac{x^2+2t^2+z}{2}\right)=-3t^4.$$ Since the two bracketed factors, $L$ and $M$ say, differ by the integer $z$ and have integer product, they are both integers. Furthermore, if $q$ is a prime common factor of $L$ and $M$, then $q$ would be a factor of both $z$ and $t$, a contradiction.

Therefore $\{L,M\}=\{au^4,cv^4\}$, where $ac=-3$ and $t=uv$, with $u$ and $v$ coprime and $uv$ even. Then $$au^4+cv^4=x^2+2t^2=x^2+2u^2v^2.$$ We can assume $v$ is even. Then $au^2\equiv 1\pmod 4$ and so $a=1$.

Therefore $u^4-2u^2v^2-3v^4=x^2$, an equation of form (2).

An equation of form (2)

Let $u,v,x$ be a pairwise coprime solution of $ u^4-2u^2v^2-3v^4=x^2$.

This can be rewritten, using completing the square, as $$\left (\frac{u^2-v^2-x}{2}\right )\left (\frac{u^2-v^2+x}{2}\right)=v^4.$$ The bracketed factors, $L$ and $M$, are again coprime integers.Therefore $\{L,M\}=\{aX^4,cY^4\}$, where $ac=1$ and $v=XY$, with $X$ and $Y$ coprime. Then $$X^4+X^2Y^2+Y^4=u^2,$$ an equation of form (1).

Fermat's infinite descent

We have seen that any positive integer solution $(x,y,z)$ of an equation of form (1) always leads to another positive integer solution $(X,Y,Z)$, where $Y=\frac{y}{2uX}<y$, which is impossible.