If you mix two solutions of different concentrations with each other you can easily determine the resulting concentration from the general formula for it:
$$c = \frac{n}{V}\tag1\label{concentration-general}$$
You have one solution with $c_1 = \pu{100 mM}$ and one with $c_2 = \pu{5 mM}$, therefore with $\eqref{concentration-general}$ you can determine the amount of substance for each, and the total amount of substance:
$$n_\mathrm{tot} = c_1\cdot V_1 + c_2\cdot V_2\tag2\label{amount-of-substance-total}$$
Assuming there is not volume contraction, we can simply add them to know the total volume:
$$V_\mathrm{total} = V_1 + V_2 \tag3\label{volume-total}$$
The total concentration therefore is:
$$c_\mathrm{total} = \frac{n_\mathrm{total}}{V_\mathrm{total}}
= \frac{c_1 V_1 + c_2 V_2}{V_1 + V_2}\tag4\label{concentration-total}$$
For your dilution series you will need to modify it, a $\eqref{volume-total}$ is fixed.
$$V_\mathrm{total} = V_1 + V_2 = \text{const.}$$
Therefore we modify $\eqref{concentration-total}$:
\begin{align}
c_\mathrm{total} &=
\frac{c_1 V_1 + c_2 (V_\mathrm{total} - V_1)}{V_\mathrm{total}}
\tag{4a}\label{concentration-total-mod-a}\\
c_\mathrm{total} &=
\frac{c_1 (V_\mathrm{total} - V_2) + c_2 V_2}{V_\mathrm{total}}
\tag{4b}\label{concentration-total-mod-b}\\
\end{align}
Now you have one remaining unknown, solve for it:
\begin{align}
V_1 &= \frac{c_\mathrm{total} - c_2}{c_1 -c_2} V_\mathrm{total}\\
V_2 &= \frac{c_\mathrm{total} - c_1}{c_2 -c_1} V_\mathrm{total}\\
\end{align}
Unless you have very accurate equipment, however, I do suggest using larger quantities than $\pu{4 mL}$.
\begin{array}{rrrrrr}
c_\mathrm{total}/\pu{mM} &
V_\mathrm{total}/\pu{mL} &
c_1/\pu{mM} &
c_2/\pu{mM} &
V_1/\pu{mL} &
V_2/\pu{mL} \\\hline
10 & 4 & 100 & 5 & 0.21 & 3.79 \\
20 & 4 & 100 & 5 & 0.63 & 3.37 \\
30 & 4 & 100 & 5 & 1.05 & 2.95 \\
40 & 4 & 100 & 5 & 1.47 & 2.53 \\
50 & 4 & 100 & 5 & 1.89 & 2.11 \\\hline
\end{array}
