$$B+Ai=e^{2i\pi/7}+e^{4i\pi/7}+e^{8i\pi/7}$$
$$B-Ai=e^{-2i\pi/7}+e^{-4i\pi/7}+e^{-8i\pi/7}$$
So $$B^2+A^2=(B+Ai)(B-Ai)=3+2\cos 2\pi/7 + 2\cos 4\pi/7+2\cos 6\pi/7$$
But $\cos (2k\pi/7) = \cos(-2k\pi/7)=\cos(2(7-k)\pi/7)$.
So this means that $B^2+A^2=3+\sum_{k=1}^{6}\cos(2k\pi/7)$.
But that sum $\sum_{k=0}^{6}\cos(2k\pi/7) = 0$, so you get that:
$$B^2+A^2=3+(-1)=2$$
More generally, for $p\equiv 3\pmod{4}$ a prime, let:
$$A=\sum_{k=1}^{(p-1)/2} \sin\frac{2\pi k^2 i}{p}\\
B=\sum_{k=1}^{(p-1)/2} \cos\frac{2\pi k^2 i}{p}$$
Then $A+Bi=\sum_{k=1}^{(p-1)/2} e^{2\pi k^2 i/p}$, and:
$$\begin{align}A^2+B^2&=\frac{p-1}{2}+\sum_{1\leq j\neq k\leq \frac{p-1}{2}}\cos\left(\frac{2\pi(k^2-j^2)}{p}\right)\\
&=\frac{p-1}{2}+\frac{1}{4}\sum_{1\leq j,k<p;\,j\not\equiv \pm k}\cos\left(\frac{2\pi(k^2-j^2)}{p}\right)\end{align}\tag{1}$$
For any factorization $n\equiv ab\pmod{p}$ with $a\not\equiv \pm b$, you have a pair $j,k$ so that $j+k=a,j-k=b$. For every $n$, since either $n$ or $-n$ is a square, there are exactly $2$ factorizations $n=ab$ ith $a\equiv \pm b$. There are $p-1$ total factorizations, there are $p-3$ factorizations with $a\not\equiv \pm b$.
So you get:
$$A^2+B^2=\frac{p-1}{2}+\frac{p-3}{4}\sum_{n=1}^{p-1}\cos(2\pi ni/p)=\frac{p+1}{4}$$
There's also some elementary Galois theory going on here. If $\zeta_p=e^{2\pi i/p}$. then the Galois group of $\mathbb Q[\zeta_p]$ over $p$ is $\mathbb Z/p\mathbb Z^{\times}\cong\mathbb Z/(p-1)\mathbb Z$.
The subgroup of squares modulo $p$ is of index $2$ in the Galois group. But $B+Ai$ is in the field fixed by this subgroup, so this means that $B+Ai$ be quadratic over $\mathbb Q$, and it is an algebraic integer. This means that $A^2+B^2$ must be an integer when $p\equiv 3\pmod{4}$.
If $p\equiv 1\pmod 4$, then $A=0$ and $B+Ai=B$ is a real algebraic integer of degree $2$ over $\mathbb Q$. You can use (1) to get $B^2$ in terms of $B$.
For $n$ not a square modulo $p$, you get that there are $p-1$ distinct factorizations $n=ab$ with $a\not\equiv -b$. For $n$ a square modulo $p$, you get that there are $p-1$ solutions to $n=ab$, but $4$ of them have $a\equiv \pm b\pmod p$, so there are $p-5$ terms for $n$. This means:
$$B^2=\frac{p-1}{2}+\frac{p-1}{4}\sum_{n=1}^{p-1}\cos\frac{2\pi ni}{p} - \frac{4}{4}B$$
or:
$$B^2=\frac{p-1}{4}-B$$
This gives that $B=\frac{-1\pm\sqrt{p}}{2}$.
Not sure which root $B$ is in general, but it is $\frac{-1+\sqrt{p}}{2}$ for small $p\equiv 1\pmod{4}$.
\Bigg( \cos(\dfrac{6\pi}{7}) + \cos(\dfrac{10\pi}{7}) + \cos(\dfrac{12\pi}{7}) \Bigg)
\ \Longrightarrow 2\Bigg( \cos(\dfrac{2\pi}{7}) + \cos(\dfrac{4\pi}{7}) + \cos(\dfrac{8\pi}{7}) \Bigg)+1=0 \Longrightarrow 2B+1=0 \Longrightarrow B=\dfrac{-1}{2}.$$
– Davood Aug 28 '17 at 17:21