Hint: $a^2 -ab +b^2 = (a+b)^2 -3ab.$
I know we can say that there exists an $x,y$ such that $ax + by = 1$. So in this case, $(a+b)x + ((a+b)^2 -3ab)y =1.$
I thought setting $x = (a+b)$ and $y = -1$ would help but that gives me $3ab =1.$ Any suggestions?
$\gcd(a+b, a^2 - ab + b^2) = \gcd(a+b, (a+b)^2 - 3ab) = \gcd(a+b, 3ab)$
Since $\gcd(a,b) = 1$ hence $\gcd(a+b,a) = 1$ and $\gcd(a+b, b)$ = 1.
Thus, $\gcd(a+b, 3ab) = \gcd (a+b, 3) $ by applying the Lemma below.
Hence, $\gcd(a+b, a^2 -ab + b^2) = \gcd(a+b, 3)=1$ or 3. (This is very simple to evaluate for given values.)
Lemma If $\gcd(m,n) = 1$, then $\gcd (m, np) = \gcd(m,p)$.
Proof: This is obvious when you consider GCD via prime factorization.
We have $\gcd(a+b,a^2-ab+b^2)=\gcd(a+b,3ab)$. If $p$ is a prime with $p|\gcd(a+b,3ab)$, then $p=3$ or $p|a$ or $p|b$. If $p|a$, then $p\nmid b$, hence $p\nmid a+b$. Similarly, if $p|b$, then $b\nmid a+b$. Thus we conclude $p=3$. But then if $9|\gcd(a+b,3ab)$, clearly $3|a$ or $3|b$ and again $3\nmid a+b$.
Hint $\,\ c\mid a\!+\!b,\overbrace{a^2\!-\!ab\!+\!b^2}^{\large f(b)}\Rightarrow\,c\mid 3a^2,3ab,3b^2\Rightarrow\,c\mid(3a^2,3ab,3b^2)=3(a,b)^2 = 3 $
by $\,\ {\rm mod}\ \ a\!+\!b\!:\ \ \color{#c00}{b\equiv -a}\,\Rightarrow\, f(\color{#c00}b)\equiv f(\color{#c00}{-a})\equiv 3a^2\equiv 3a(-b)\equiv 3(-b)(-b) $
A solution and some generalization: First of all, we can replace $b$ by $-b$, since $\gcd(a,b)=\gcd(a,-b)$. So we need to show $\gcd(a^2+ab+b^2,a-b)$.
If $p$ is a prime dividing the gcd, then it also divides $a-b$ which means $a \equiv b \mod p$, hence: $$0 = a^2+ab+b^2=a^2+a^2+a^2=3a^2 \mod p$$ So either $p|3$ or $p|a$. $p$ can't divide $a$, because then it also divides $b$ and so $\gcd(a,b) \neq 1$. Thus $p=3$. So $\gcd(a^2+ab+b^2,a-b)$ is a power of 3 (this includes 1).
Can it be divisible by 9? No, since $a^2+ab+b^2=(a-b)^2+3ab$, so if $9$ divides the gcd, it divides $(a-b)^2$ and $a^2+ab+b^2$, and so it divides $3ab$, i.e. $3|ab$, so one of $a,b$ is divisible by 3, a contradiction ($3|a-b,a\implies 3|b \implies (a,b)!= 1$).
The proof resembles Hagen von Eitzen's proof, but it can be generalized. What is very noticeable is that $a^2+ab+b^2,a-b$ are both homogenizations of Cyclotomic polynomials, $\phi_1(x)=x-1$ and $\phi_3(x)=x^2+x+1$.
The following more general statement can be proved in a similar way: let $\phi_n(x), \phi_m(x)$ be 2 Cyclotomic polynomials. Assume $m=1$ and $n=p$, a prime, so $\phi_m(x)=x-1,\phi_n(x)=x^{p-1}+x^{p-2} + \cdots + 1$.
Define their homogenization as $h_n(x,y)=\phi_n(\frac{x}{y})y^{\phi(n)}=\sum_{i=0}^{p-1}x^i y^{p-1-i},h_m(x,y)=\phi_m(\frac{x}{y})y^{\phi(m)}=x-y$.
I claim that $\gcd(x,y)=1 \implies \gcd(\phi_m(x,y),\phi_m(x,y)) \in \{1,p \}$.
Proof (for $p \neq 2$):
If a prime $q$ divides the gcd, then $\sum_{i=0}^{p-1}x^i y^{p-1-i} = px^{p-1} \equiv 0 \mod q \implies p=q$, as before.
$p^2$ can't divide the gcd: assume $x \equiv y \mod p$, i.e. $x=y+pk$ for some integer $k$. Note that $x^i = y^i + y^{i-1}pki \mod p^2$. Thus: $$\sum_{i=0}^{p-1}x^i y^{p-1-i} = \sum_{i=0}^{p-1} ( y^i + y^{i-1}pki) y^{p-1-i} =$$ $$p y^{p-1} + pk y^{p-2} \sum_{i=1}^{p-1} i = py^{p-1} + pk y^{p-2} \binom{p}{2} =$$ $$ y^{p-2}p (y+k\frac{p-1}{2}) \mod {p^{2}}$$
Since $p$ doesn't divide $y$ (as $\gcd(x,y)=1$), if this is zero modulo $p^2$ it means $y+k\frac{p-1}{2} \equiv 0 \mod p$, and so $p \nmid k$ (else $p|y$, contradiction), so $p^2 \nmid x-y$.
A new question arises - what are the possible values of $gcd(\phi_n(x), \phi_m(x))$, for general $n,m$? From this problem I can probably deduce a similar result for homogeneous polynomials.
EDIT: I think that the fact $\phi_n(1) = p$ if $n=p^{k}$ and $1$ otherwise should help, it gives $\gcd(h_n(x,y),h_m(x,y))=1$ if $(n,m)=1$ and one of $n,m$ is not a prime power or 1, and $\gcd(h_n(x,y),h_m(x,y))$ is a power of $p$ if $n=1, m=p^k, k>1$.
Suppose that the positive integer $d$ divides both $a+b$ and $a^2-ab+b^2$. You showed that $d$ divides $3ab$.
Note that $d$ is relatively prime to $ab$. Otherwise, there is a prime $p$ that divides both $d$ and $ab$. Since $p$ divides $ab$, it follows that $p$ divides one of $a$ or $b$, say $a$. But since $p$ divides $d$, it follows that $p$ divides $a+b$. Since $p$ divides $a$, it follows that $p$ divides $b$. This contradicts the fact that $a$ and $b$ are relatively prime.
Since $d$ divides $3ab$, and $d$ is relatively prime to $ab$, we conclude that $d$ divides $3$. So if $d$ is positive, then $d=1$ or $d=3$.
Note that both $1$ and $3$ are possible. For let $a=1$ and $b=1$. Then $\gcd(a+b,a^2-ab+b^2)=1$.
Now let $a=2$ and $b=1$. Then $a^2-ab+b^2=3$, so in this case the gcd is $3$.
$\gcd(a+b, a^2-ab+b^2) = \gcd(a+b, (a+b)^2-3ab) = \gcd(a+b, 3ab)$ by the Euclidean algorithm. If the gcd was $d \ne 1 ,3$, then $d \mid a$ or $d \mid b$ in $3ab$ but then from $a+b$, $d$ would divide the other. Thus, the result follows.
This is my solution:
Lemma 1: $\forall a,b\in\mathbb{Z}$, if $\gcd(a,b)=1\Rightarrow\gcd(a+b,a)=\gcd(a+b,b)=1$.
Lemma 2: $\forall a,b,c\in\mathbb{Z}$, if $\gcd(a,b)=1\Rightarrow\gcd(a,bc)=\gcd(a,c)$.
Proving lemma 1:
Let $d=\gcd(a+b,a)\Rightarrow d\mid(a+b)~\wedge~d\mid a\Rightarrow d\mid b$, but $\gcd(a,b)=1\Rightarrow d=1$.
Let $d=\gcd(a+b,b)\Rightarrow d\mid(a+b)~\wedge~d\mid b\Rightarrow d\mid a$, but $\gcd(a,b)=1\Rightarrow d=1$.
Proving lemma 2:
Let $d=\gcd(a,bc)\Rightarrow d\mid a~\wedge~d\mid bc\Rightarrow d\mid ac~\wedge~d\mid bc\Rightarrow d\mid\gcd(ac,bc)\Rightarrow d\mid c\cdot\gcd(a,b)\Rightarrow d\mid c$.
Then $d\mid a~\wedge~d\mid c\Rightarrow d\mid\gcd(a,c)\Rightarrow d\le\gcd(a,c)$.
Let $d'=\gcd(a,c)\Rightarrow d'\mid a~\wedge~d'\mid c\Rightarrow d'\mid a~\wedge~d'\mid bc\Rightarrow d'\mid\gcd(a,bc)\Rightarrow d'\le\gcd(a,bc)$.
Then $\gcd(a,bc)\le\gcd(a,c)~\wedge~\gcd(a,c)\le\gcd(a,bc)\Rightarrow\gcd(a,bc)=\gcd(a,c)$.
Proving the exercise:
Let's prove this: $\forall a,b\in\mathbb{Z},~\gcd(a,b)=1\Rightarrow\gcd(a+b,a^2+b^2-nab)\mid(n+2)$
Let $d=\gcd(a+b,a^2+b^2-nab)$
$\Rightarrow d\mid(a+b)~\wedge~d\mid(a^2+b^2-nab)$
$\Rightarrow d\mid(a+b)^2~\wedge~d\mid(a^2+b^2-nab)$
$\Rightarrow d\mid\Big((a+b)^2-(a^2+b^2-nab)\Big)$
$\Rightarrow d\mid(a^2+b^2+2ab-a^2-b^2+nab)$
$\Rightarrow d\mid ab(n+2)$
Suppose $d\mid a~\vee~d\mid b\Rightarrow d\mid ab\Rightarrow d\mid(a+b)~\wedge~d\mid ab\Rightarrow d\mid\gcd(a+b,ab)$
$\gcd(a+b,a)=1$ (Lemma 1)
$\gcd(a+b,ab)=\gcd(a+b,b)=1$ (Lemma 2, Lemma 1)
Then $d\mid1\Rightarrow d=1\Rightarrow d\mid(n+2)$.
If $d\nmid ab\Rightarrow d\mid(n+2)$
Then the value of $d$ doesn't matter, $d\mid(n+2)$.
In this case; $d=\gcd(a+b,a^2+b^2-ab),~n=1\Rightarrow d\mid(1+2)\Rightarrow d\mid3\Rightarrow d=1~\vee~d=3.$
We will do this using only divisibility. Implying that you have this knowledge.
$(a+b,a^2-ab+b^2)=\{1\;\text{and}\;3\}$ if $(a,b)=1$
Calling $d$ the $(a+b,a^2-ab+b^2)$ $$(a+b,a^2-ab+b^2)=d\Longrightarrow d\mid a+b\;\;\text{and}\;\;d\mid a^2-ab+b^2$$$$a^2-ab+b^2=a^2-ab+b^2\underbrace{+3ab-3ab}_{=0})=a^2+2ab+b^2-3ab=(a+b)^2-3ab$$If $d\mid a+b\Longrightarrow d\mid (a+b)^2$ as $d\mid (a+b)^2-3ab$ For these data we have $d\mid3ab$
There are three possibilities
I) $d\mid 3\Longrightarrow d=1$ or $d=3$
II) $d\mid a$ Above we see that $d\mid a+b$ this implies that $d\mid b\Longrightarrow d\mid(a,b)\Longrightarrow d\mid1\Longrightarrow d=1$
III) $d\mid b$ Above we see that $d\mid a+b$ this implies that $d\mid a\Longrightarrow d\mid(a,b)\Longrightarrow d\mid1\Longrightarrow d=1$
Thus, we see that the unique possibilities for $d$ is $1$ or $3$.
Let $d=(a+b,\,a^2-ab+b^2)$. This means $d\,|\,(a+b)$ and $d\,|\,(a^2-ab+b^2)$. Thus, $d\,|\,\{(a+b)^2-(a^2-ab+b^2)\}$ or $d\,|\,3ab$.
Claim: $(d,\,ab)=1$.
Suppose, not. Then $e=(d,\,ab)>1$. Thus, $p$ be a prime such that $p\,|\,e$; which means $p\,|\,d\implies p\,|\,(a+b)$. Moreover, $p\,|\,ab\implies p\,|\,a\,$ or $\,p\,|\,b$. Without loss of generality, suppose $p\,|\,a$. Thus, $p\,|\,(a+b-a)=b$. Thus, $p\,|\,(a,\,b)$. A contradiction as $(a,\,b)=1$, which proves our claim.
Thus, we have $d\,|\,3ab$ and $(d,\,ab)=1$. This implies $d\,|\,3$. Therefore, $d=1$ or $d=3$
