I am not sure how to tackle this problem and yes it is a homework problem. Here is what I have. I know that $\text{ord}(a) = h$ and h is even so $h = 2 \alpha$ for some $\alpha \in \mathbb{Z}$. I also know Euler's criterion which states that $$a^{\frac{p-1}{2}} = -1 \pmod{p}$$ if there are no integers $x$ which satisfy $x^2 = a \pmod{p}$. So my steps are
$a^h = (x^2)^h = 1 \pmod{p}$ which implies $p | x^{2h} - 1$. I am now stuck, am I in the right direction? Please let me know a hint.
Thanks
The problem must be to show $\,a^{h/2}=-1\pmod p\,$ , but this follows almost immediately from:
$$ord_p(a)=h\Longleftrightarrow a^h=1\pmod p\,\,\,\wedge\,\,\,a^m\neq 1\,\,\,\forall\,m<h\Longrightarrow $$
$$a^{h/2}\,\,\,\text{is a square root of 1 different from 1 itself}\Longrightarrow a^{h/2}=-1$$
since the only square roots of unit in any field are $\,\pm 1\,$ (and not in any case they're different...)
Hint $\rm\ mod\ p\!:\ 0 \equiv a^h\!-1\equiv (a^{h/2}\!-1)(a^{h/2}\!+1)\Rightarrow a^{h/2}\!+1 \equiv 0,\:$ by cancelling $\rm\,a^{h/2}\!-1$ $(\not\equiv 0,\,$ else the order of $\rm\,a\,$ is $\rm\le h/2).\:$ Elements $\ne 0$ are invertible (so cancellable) because $\rm\,\Bbb Z/p\,$ is a field.
Remark $\ $ With $\rm\:x = a^{h/2},\:$ the above proof shows that $\rm\,x\not\equiv 1,\ x^2\equiv 1\:\Rightarrow\: x\equiv -1,\:$ i.e. the only square roots of $1$ are $\pm 1.\,$ This is a special case of the fact that of the fact that a nonzero polynomial over a field (or domain) has no more roots than its degree (a property which characterizes integral domains among rings). Indeed, some algorithms for factoring an integer $\rm\,n\,$ work by searching $\rm\,mod\ n\,$ for a square root of $1$ that is nontrivial, i.e. $\ne \pm1.\:$ See here for further discussion on such.
