Note that $[F(\alpha):F]=n$. Also, from Dedekind’s Product Theorem,
$$ n=[F(\alpha):F]=[F(\alpha):F(\alpha^m)][F(\alpha^m):F].\tag{1}$$
Write $n=mk$. If $g(x)$ is the minimal polynomial for $\alpha^m$ over $F$, and has degree $r$, then $g(x^m)$ is a polynomial that is satisfied by $\alpha$, and hence a multiple of $x^n-b$. That means that $n\leq\deg(g(x^m)) = m\deg(g) = mr$. Therefore, $r\geq k$. Thus, the minimal polynomial of $\alpha^m$ is of degree at least $k$. But clearly, $\alpha^m$ satisfies the polynomial $x^k - b$, so this is the minimal polynomial for $\alpha^m$ over $F$. Thus, $[F(\alpha^m):F]=k = \frac{n}{m}$.
Plugging into (1), we get
$$ n = [F(\alpha):F(\alpha^m)]k,$$
so $[F(\alpha):F(\alpha^m)] = m$.
Since $\alpha$ clearly satisfies $x^m-\alpha^m$, and this is monic of the correct degree, it must be the minimal polynomial for $\alpha$ over $F(\alpha^m)$.