First, let's see why the degree-genus formula fails. Remember that the degree-genus formula tells you the arithmetic genus (i.e., $h^1(X,\mathcal{O}_X)$), but if the curve is singular, this may not be equal to the geometric genus $h^0(X,\omega_{X/k}).$
Considered as a projective plane curve in the usual way (via homogenization), your curve $V(X^4 + Z^4 - Y^2Z^2)$ is singular at infinity. Let's use the Jacobian criterion: we see that
\begin{align*}
\frac{\partial F}{\partial X} &= 4X^3\\
\frac{\partial F}{\partial Y} &= -2YZ^2\\
\frac{\partial F}{\partial Z} &= 4Z^3 - 2Y^2Z.
\end{align*}
At $[0:1:0],$ all of these partials vanish, so that the curve is not smooth at $\infty,$ and hence the degree-genus formula will not necessarily compute the geometric genus of $V(X^4 + Z^4 - Y^2Z^2)$ without modification (see in particular Hartshorne exercise IV.1.8).
Now, let's compute the (geometric) genus, considering instead of the model $V(X^4 + Z^4 - Y^2Z^2),$ the corresponding nonsingular projective curve $X.$ There's a double cover $\pi : X\to\Bbb P^1_k$ of $\Bbb P^1_k$ by this curve given by $[x: y: z]\mapsto [x : z],$ so we can use the Riemann-Hurwitz formula. This tells us
\begin{align*}
2h^0(X,\omega_X) - 2 &= 2(2h^0(\Bbb P^1_k,\mathcal{O}(-2)) - 2) + \deg R\\
\implies 2h^0(X,\omega_X) - 2 &= -4 + \deg R\\
\implies h^0(X,\omega_X) &= -1 + \frac{1}{2}\deg R,
\end{align*}
where $R = \sum_{P\in X} (e_P - 1)P$ is the ramification divisor of $X.$ Because the morphism to $\Bbb P^1$ is a double cover, the ramification index $e_P$ of any point is at most $2.$ In particular, if $P$ is ramified, then $e_P = 2,$ and otherwise $e_P = 1.$ The ramified points on $X$ are precisely the points $[x : y : z]$ such that $y = 0$ (these are the points with only one preimage under our map $\pi$). There are four of these (the four roots of $x^4 + 1$), so we have
$$
h^0(X,\omega_X) = -1 + \frac{4}{2} = 2 - 1 = 1.
$$
In computing the geometric genus, I assumed that $X$ was smooth. When one refers to the curve $y^2 = x^4 + 1$ (or any hyperelliptic curve given by an equation of the form $y^2 = f(x)$), one often implicitly means the nonsingular projective curve corresponding to this affine curve. Again, as we've shown, this is not simply $V(X^4 + Z^4 - Y^2Z^2)$: this plane curve is singular. Using weighted projective space, or by gluing affine models, you can obtain the desired nonsingular curve, see here for example.