As you said, the desired probability is $\frac{1}{4^{10}}$.
You want to count the solutions of the inequality
$$x_1 + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} \leq 39 \tag{1}$$
subject to the restrictions $x_j \leq 4$ for $1 \leq j \leq 10$.
If we let
$$s = 40 - (x_1 + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10})$$
then a solution of inequality 1 in the positive integers is a solution of the equation
$$x_1 + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} + s = 40 \tag{2}$$
in the positive integers. As you correctly found, there are
$$\binom{40 - 1}{11 - 1} = \binom{39}{10}$$
such solutions. From these, we wish to exclude those solutions in which at least one of the variables $x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9, x_{10}$ exceeds $4$.
Suppose $x_1 \geq 5$. Let $x_1' = x_1 - 4$. Then $x_1$ is a positive integer. Substituting $x_1' + 4$ for $x_1$ in equation 2 yields
\begin{align*}
x_1' + 4 + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} & \leq 39\\
x_1' + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} & \leq 35 \tag{3}
\end{align*}
Let
$$s' = 36 - (x_1' + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10})$$ Then the number of solutions of inequality 3 in the positive integers is equal to the number of solutions of the equation
$$x_1' + x_2 + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} + s' = 36 \tag{4}$$
in the positive integers. Equation 4 has
$$\binom{36 - 1}{11 - 1} = \binom{35}{10}$$
solutions in the positive integers. By symmetry, there would be an equal number of solutions for each of the ten variables in inequality 3 that could exceed $4$. Hence, we have
$$\binom{10}{1}\binom{35}{10}$$
solutions in which one of the variables exceeds $5$. Subtracting these from the total gives us your answer
$$\binom{39}{10} - \binom{10}{1}\binom{35}{10}$$
So what went wrong?
We have subtracted those cases in which two of the variables exceed $4$ twice, once for each way we could designate one of the variables as the variable that violates the restriction that the variable not exceed $4$. Therefore, we need to add them back.
Suppose $x_1 \geq 5$ and $x_2 \geq 5$. Let $x_1' = x_1 - 4$; let $x_2' = x_2 - 4$. Then $x_1'$ and $x_2'$ are positive integers. Substituting $x_1' + 4$ for $x_1$ and $x_2' + 4$ for $x_2$ in inequality 1 and simplifying yields
$$x_1' + x_2' + x_3 + x_4 + x_5 + x_6 + x_7 + x_8 + x_9 + x_{10} \leq 31 \tag{5}$$
Inequality 5 is an inequality in the positive integers with
$$\binom{32 - 1}{11 - 1} = \binom{31}{10}$$
solutions. By symmetry, we would obtain the same number of solutions for each of the $\binom{10}{2}$ ways two of the variables could exceed $4$. Hence, there are
$$\binom{10}{2}\binom{31}{10}$$
solutions in which two of the variables exceed $4$.
Thus far, we have
$$\binom{39}{10} - \binom{10}{1}\binom{35}{10} + \binom{10}{2}\binom{31}{10}$$
However, we have not excluded solutions in which three of the variables exceed $4$ since we first subtracted three times, once for each of the three ways we could designate one of those variables as the one that exceeds $4$, and added them three times, once for each of the $\binom{3}{2}$ ways we could designate two of those variables as the ones that exceed $4$. Therefore, we need to subtract those from the total.
The desired count can be found by using the Inclusion-Exclusion Principle.
$$\binom{39}{10} - \binom{10}{1}\binom{35}{10} + \binom{10}{2}\binom{31}{10} - \binom{10}{3}\binom{27}{10} + \binom{10}{4}\binom{23}{10} - \binom{10}{5}\binom{19}{10} + \binom{10}{6}\binom{15}{10} - \binom{10}{7}\binom{11}{10} = 1~048~575 = 4^{10} - 1$$
However, there is an easier method. Let $y_j = 5 - x_j$ for $1 \leq j \leq 10$. Then each $y_j$ is a positive integer satisfying $1 \leq y_j \leq 4$. Substituting $5 - y_j$ for $x_j$, $1 \leq j \leq 10$, in inequality 1 yields
\begin{align*}
50 - y_1 - y_2 - y_3 - y_4 - y_5 - y_6 - y_7 - y_8 - y_9 - y_{10} & \leq 39\\
-y_1 - y_2 - y_3 - y_4 - y_5 - y_6 - y_7 - y_8 - y_9 - y_{10} & \leq -11\\
y_1 + y_2 + y_3 + y_4 + y_5 + y_6 + y_7 + y_8 + y_9 + y_{10} & \geq 11 \tag{6}
\end{align*}
The number of solutions of inequality 6 is found by subtracting the number of solutions of the inequality
$$y_1 + y_2 + y_3 + y_4 + y_5 + y_6 + y_7 + y_8 + y_9 + y_{10} \leq 10 \tag{7}$$ in the positive integers from $4^{10}$. There is one such solution, so we obtain $4^{10} - 1$ solutions for inequality 1.