In my calculus book (Calculus: A Rigorous First Course), I introduced notation to make it easier to explain what's going on here. $\lim_{x \to a} g(x) = b$ means that if $x$ is close to $a$, but not equal to $a$, then $g(x)$ is close to $b$ (with the details of what "close" means being spelled out by the $\epsilon$-$\delta$ formalism). This is sometimes expressed by writing "as $x \to a$, $g(x) \to b$." But this notation leaves out the important fact that $x$ cannot be equal to $a$. So in my book I changed the notation slightly to "as $x \to a^\ne$, $g(x) \to b$," with the superscript "$\ne$" indicating that $x$ cannot be equal to $a$. The notation is modeled on other common notation like "as $x \to a^+$, $g(x) \to b$," which means that if $x$ is close to $a$ but larger than $a$, then $g(x)$ is close to $b$.
Using this notation, we can see more clearly what the issue is. I will follow the suggestion, from Ryszard Szwarc's comment, that it is helpful to let $y = g(x)$. Then $f(g(x)) = f(y)$, and the question is: if $\lim_{x \to a} g(x) = b$, does it follow that $\lim_{x \to a} f(g(x)) = \lim_{y \to b} f(y)$? Rewriting the question using my modified arrow notation, the question is: Suppose that as $x \to a^\ne$, $y = g(x) \to b$, and as $y \to b^\ne$, $f(y) \to L$. Does it follow that as $x \to a^\ne$, $f(g(x)) = f(y) \to L$? The notation now points out the issue: as $x \to a^\ne$, we have $y \to b$, but we need to have $y \to b^\ne$ to conclude that $f(g(x)) = f(y) \to L$. It is the mismatch between $y \to b$ and $y \to b^\ne$ that is the problem.
There are multiple ways to fix the problem. One is, as suggested by Andrew D. Hwang's answer, to assume that when $x$ is close to $a$, $g(x)$ is never equal to $b$. That would allow us to say that as $x \to a^\ne$, $y = g(x) \to b^\ne$. This could then be combined with the fact that as $y \to b^\ne$, $f(g(x)) = f(y) \to L$ to conclude that as $x \to a^\ne$, $f(g(x)) \to L$. The justification for stringing together the arrow notations is given by the $\epsilon$-$\delta$ argument in Andrew D. Hwang's answer.
But another possibility is to assume that $f$ is continuous at $b$. In
that case, if $\lim_{y \to b} f(y) = L$, then we can say that as $y \to b$, $f(y) \to L$; there is no need for the superscript "$\ne$" on $b$ because continuity tells us that when $y = b$, $f(y) = L$. Now we can string together the facts that as $x \to a^\ne$, $y = g(x) \to b$ and as $y \to b$, $f(g(x)) = f(y) \to L$ to conclude that as $x \to a^\ne$, $f(g(x)) \to L$, or in other words, $\lim_{x \to a} f(g(x)) = L$. Once again, it is easy to justify this with $\epsilon$-$\delta$ reasoning.
And there are lots of other possibilities that can be expressed with this notation. For example, suppose we know that as $x \to a^\ne$, $y = g(x) \to b^-$ (that is, if $x$ is close to $a$ but not equal to $a$, then $g(x)$ is close to $b$ and less than $b$). Then we can compute $\lim_{y \to b^-} f(y)$. If the answer is $L$, then we can say that as $y \to b^-$, $f(y) \to L$, and we can string together the two limit statements to conclude that as $x \to a^\ne$, $f(g(x)) \to L$, and therefore $\lim_{x \to a} f(g(x)) = L$.
My conclusion is that the usual arrow notation for describing limits is misleading because it leaves out an important part of the meaning of limits. But when this deficiency is fixed, the notation can become a reliable guide to how limits can be combined to compute limits of composite functions.