I'm a high school teacher and someone asked me this in my class, and to be honest I'm quite stumped! I haven't done any high level math in such a long time, and I'm really not sure how to approach this.
Is the solution even approachable to a highschool student?
Thanks for the help.
It is not possible, and the proof can be done with some trigonometry and a basic understanding of rational numbers:
Suppose the angle between two vectors $(a,b)$ and $(c,d)$ in the plane is $\alpha - \beta$, where $\alpha$ is the larger of the two angles made by the two vectors with the $x$-axis, and $\beta$ is the smaller.
Then
$$\tan(\alpha-\beta) = \frac{\tan \alpha - \tan \beta}{1+\tan \alpha \tan \beta} = \frac{ad-bc}{ac+bd}$$
Thus the $\tan$ of any angle made between two integer lattice vectors is a rational number.
By dividing a regular octagon into 8 isosceles triangles and looking at the base angles of these 8 triangles, we see that we need an angle of $3\pi/8$, but $\tan (3\pi/8) = 1+\sqrt{2}$ which is irrational. Thus the octagon is not possible.
Note 1:
The angle $3\pi/8$ which is used in the proof, is actually the angle between one side of the octagon, and a "diameter" of the octagon, joining two opposite vertices. Clearly, the end-point of these two lengths would have integer coordinates if such a regular (integer) octagon were possible.
Note 2:
It seems (judging by the comment below) that my explanation has not been clear enough for everyone. I am NOT talking about the angle between two adjacent sides of an octagon being $3\pi/8$ here. The angle between adjacent sides is $3\pi/4$, and the angle I am talking about is exactly half of that.
Edit:
This question and the answer by André Nicolas give more information - and an alternative proof.
Consider the two adjacent sides of a regular octagon pictured below
$\hspace{5cm}$
First, since the sides have equal length, we have $$ \begin{align} (c-b)\cdot(b-a) &=|c-b|\,|b-a|\cos(\pi/4)\\ &=\frac{|b-a|^2}{\sqrt2} \end{align} $$ Thus, we have $$ \sqrt2=\frac{(b-a)\cdot(b-a)}{(c-b)\cdot(b-a)} $$ If all the coordinates are rational, then the quantity on the right is rational. However, since $\sqrt2$ is not rational, that is impossible.
When such an octogon exists then you are able to find a lattice point $(a,b)$ in the first quadrant and a lattice point $(c,d)$ with $d<0$ such that (i) $a^2+b^2=c^2+d^2$ and (ii) the enclosed angle is $135^\circ$.
Turning $(a,b)$ by ninety degrees counterclockwise produces the point $(-b,a)$ in the second quadrant. The condition (ii) then implies that there is a real $\lambda>0$ with $$(c,d)=-\lambda\bigl((a,b)+(-b, a)\bigr)\ .\tag{1}$$ From $\lambda={-d\over a+b}$ we conclude that in fact $\lambda\in{\mathbb Q}$.
Using the condition (i) and $(1)$ we now obtain $$a^2+b^2=c^2+d^2=2\lambda^2(a^2+b^2)\ ,$$ or $2\lambda^2=1$, which is incompatible with $\lambda\in{\mathbb Q}$.
There is a reasonably simple argument that shows that this is impossible for any regular $n$-gon other than a square (although the equilateral triangle and hexagon need a bit of extra work since these can appear on a hexagonal lattice).
The argument goes like this. Suppose there are grid points forming a regular octagon. Let $A, B, C$ be three successive vertices. Then form $D = A-B+C$ (such that $ABCD$ forms a rhombus). Now $D$ is a grid point located inside the octagon and it is not its center. Repeating this construction for the other vertices hence results in another regular octagon with vertices on grid points that is strictly smaller. This leads to an ever shrinking set of such octagons, which is not possible.
In other words, there can be no smallest such octagon and therefore there can be none at all.
