[The following solution relies on the ideas in the previous two answers.]
Let the roots of $x^4+ax^3+bx^2+ax+1=0$ be given by $c, \frac{1}{c}, d, \frac{1}{d}$ where $c\in\mathbb{R}$, and
let $t=c+\frac{1}{c}$ and $l=d+\frac{1}{d}$.
Then $|t|\ge2$ since $c+\frac{1}{c}\ge2$ if $c>0$ and $c+\frac{1}{c}\le2$ if $c<0$.
Since $x^4+ax^3+bx^2+ax+1=(x-c)(x-\frac{1}{c})(x-d)(x-\frac{1}{d})=(x^2-tx+1)(x^2-lx+1)$,
we have that $a=-(t+l)$ and $b=2+lt$; so
$a^2+b^2=(t+l)^2+(2+lt)^2=t^2+2lt+l^2+4+4lt+l^2t^2=(l^2+1)t^2+6lt+l^2+4$.
We consider two cases:
$\mathbf{A)}$ If $d\in\mathbb{R}$, then $|l|\ge2$ (for the same reason that $|t|\ge2$); so
$\;\;\;\;a^2+b^2=(l^2+1)t^2+6lt+l^2+4\ge20+6lt+8\ge4$ $\;$(since $|lt|\le4\implies lt\ge-4$).
$\mathbf{B)}$ If $d\not\in\mathbb{R}$, then $\frac{1}{d}=\bar{d}\implies|d|=1$, so $d=e^{i\theta}=\cos\theta+i\sin\theta$ for some $\theta\in\mathbb{R}$. Then
$\;\;\;\;l=d+\frac{1}{d}=d+\bar{d}=2\cos\theta$, so $|l|\le2$.
With $l$ fixed, let $f(t)=a^2+b^2=(l^2+1)t^2+6lt+l^2+4$.
Since the vertex of the graph of $f$ is at $t=\frac{-3l}{l^2+1}$,
and $-2<\frac{-3l}{l^2+1}<2\;\;\;\;$ (since $2l^2+3l+2>0$ and $2l^2-3l+2>0$),
the minimum value of f for $|t|\ge2$ will occur when $t=2$ or $t=-2$:
1) If $t=2$, $a^2+b^2=4(l^2+1)+12l+l^2+4=5l^2+12l+8$, and
$\;\;\;g(l)=5l^2+12l+8$ has its minimum when $l=-\frac{6}{5}$, with $g(-\frac{6}{5})=\frac{4}{5}$.
$\;\;\;\;$(Notice that $|-\frac{6}{5}|\le2$).
2) If $t=-2$, $a^2+b^2=4(l^2+1)-12l+l^2+4=5l^2-12l+8$, and
$\;\;\;g(l)=5l^2-12l+8$ has its minimum when $l=\frac{6}{5}$, with $g(\frac{6}{5})=\frac{4}{5}$.
$\;\;\;\;$(Notice that $|\frac{6}{5}|\le2$).
From parts A and B, we can conclude that the minimum value of $a^2+b^2$ is $\;\;4/5$.