How about something like this:
Define:
$$ \begin{align*}
R_0 &= 1 \\
\alpha_0 &=0 \\
\Xi (i, j;N) &\equiv ({R_i\over R_j}{R_{1}R_{2}\dots R_i \dots R_N})
\cos({-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k}})
\\
&\equiv ({R_i\over R_j}{\prod_{k}^{N}{R_k}})\cos(-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k})
\end{align*}
$$
The reason for $\equiv$ rather than $=$ is to establish it as an identity so even if $R_j = 0$ the identity holds.
So,
$$
\begin{align*}
\Xi(1,0;3) &\equiv
{{R_1}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_1 + \sum_{k=1}^{3}{\alpha_k})
&= R_1^2R_2R_3 \cos(-2\alpha_1 +\alpha_2+\alpha_3) \\
\Xi(2,0;3) &\equiv
{{R_2}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_2 + \sum_{k=1}^{3}{\alpha_k})
&= R_1R_2^2R_3 \cos(\alpha_1-2\alpha_2+\alpha_3) \\
\Xi(3,0;3) &\equiv
{{R_3}\over{R_0}}{R_1R_2R_3}\cos(-\alpha_0 - 3\alpha_3 + \sum_{k=1}^{3}{\alpha_k})
&= R_1R_2R_3^2 \cos(\alpha_1-2\alpha_2+\alpha_3) \\
\quad &\vdots \\
\Xi(2,3;4) &\equiv {{R_2}\over{R_3}}{R_1R_2R_3R_4}\cos(-\alpha_3 - 3\alpha_2 + \sum_{k=1}^{4}{\alpha_k})
&= R_1R_2^2R_4\cos(\alpha_1-2\alpha_2+\alpha_4)
\end{align*}$$
Now, let's map your $M$s:
$$
\begin{align*}
M_3 = \Xi(1,0;3) + \Xi(2,0;3) + \Xi(3,0;3) = \sum_m{\Xi(m, 0; 3)}
\end{align*}
$$
$$
\begin{align*}
M_4 = \quad &\Xi(1,4;4) + \Xi(2,4;4) + \Xi(3,4;4)
&(= R_1^2R_2R_3\cos(-2\alpha_1+\alpha_2+\alpha_3)+\dots) \\
+&\Xi(1,3;4) + \Xi(2,3;4) + \Xi(4,3;4) \\
+&\Xi(4,2;4) + \Xi(1,2;4) + \Xi(3,2;4) \\
+&\Xi(4,1;4) + \Xi(3,1;4) + \Xi(2,1;4) \\
\end{align*}
$$
Since I don't have the $M_5$ example, I cannot really be definitive. However there's a pattern emerging for $\Xi(\_,\_;4)$ that can be converted to a more compact form.
Rearranging the terms above a little:
$$
\begin{align*}
M_4 = \quad
&&+\Xi(2,1;4) &+ \Xi(3,1;4) &+ \Xi(4,1;4) \\
&+\Xi(1,2;4) &&+ \Xi(3,2;4) &+ \Xi(4,2;4) \\
&+\Xi(1,3;4) &+ \Xi(2,3;4) &&+ \Xi(4,3;4) \\
&+\Xi(1,4;4) &+ \Xi(2,4;4) &+ \Xi(3,4;4)
\end{align*}
$$
Now obviously the terms that are missing are of the form $\Xi(A,A;N)$ so you can simply define $\Xi(A,A;N) \equiv 0$
Which means the definition above becomes something like:
$$
\Xi (i, j;N) \equiv \begin{cases}
({R_i\over R_j}{\prod_{k}^{N}{R_k}})\cos(-\alpha_j-3\alpha_i+\sum_{k=1}^{N}{\alpha_k}) & \forall {i\neq j}\\
0
\end{cases}
\\
R_0 = 1; \alpha_0 = 0; i=1,2,3\dots; j=1,2,3\dots; N\in\mathbb{Z_+}
$$
Given this new definition we can:
$$
\begin{align*}
M_4 = \quad&\Xi(1,1;4) &+& \Xi(2,1;4) &+& \Xi(3,1;4) &+& \Xi(4,1;4) \\
+ &\Xi(1,2;4) &+& \Xi(2,2;4) &+& \Xi(3,2;4) &+& \Xi(4,2;4) \\
+ &\Xi(1,3;4) &+& \Xi(2,3;4) &+& \Xi(3,3;4) &+& \Xi(4,3;4) \\
+ &\Xi(1,4;4) &+& \Xi(2,4;4) &+& \Xi(3,4;4) &+& \Xi(4,4;4)
\end{align*}
$$
Now we can simplify the expression by defining $M_N$ as (order of summation is changed but works)
$$
M_N = \sum_{i=1}^{N}{\sum_{j=1}^{N}{\Xi(i,j;N)}}
$$