The most transparent and general way to deal with these questions is with generating functions. The humble geometric series identity says:
$$1+x+\cdots+x^{n-1} = \frac{1-x^n}{1-x}$$
Note that $\frac{d^2}{dx^2} x^i = i(i-1) x^{i-2}$. We can get a sum of triangular number coefficients by differentiating the above twice, dividing by $2$, and setting $x=1$. You want squares of triangular numbers though. For that, just notice
$$\left(x^2 \frac{d^2}{dx^2} x^2 \frac{d^2}{dx^2}\right) x^i = x^2\frac{d^2}{dx^2} x^2 i(i-1)x^{i-2} = i^2 (i-1)^2 x^i. $$
Thus
$$\begin{align*}
\sum_{i=1}^n \left(\frac{i(i+1)}{2}\right)^2
&= \sum_{i=0}^{n+1} \left(\frac{i(i-1)}{2}\right)^2 \\
&= \sum_{i=0}^{n+1} \left.\left(\frac{i(i-1)}{2}\right)^2 x^i\right|_{x=1} \\
&= \frac{1}{4} \sum_{i=0}^{n+1} \left.\left(x^2 \frac{d^2}{dx^2} x^2 \frac{d^2}{dx^2} x^i\right)\right|_{x=1} \\
&= \frac{1}{4} \left.\left(x^2 \frac{d^2}{dx^2} x^2 \frac{d^2}{dx^2} \sum_{i=0}^{n+1} x^i\right)\right|_{x=1} \\
&= \frac{1}{4} \lim_{x \to 1} \left(x^2 \frac{d^2}{dx^2} x^2 \frac{d^2}{dx^2} \frac{1-x^{n+2}}{1-x}\right) \\
&= \cdots \\
&= \frac{n (1 + n) (2 + n) (1 + 3 n (2 + n))}{60}
\end{align*}$$
Here $\cdots$ hides the mess. A few remarks are in order:
- No special identities are required--no random Faulhaber formula, sum of squares, sum of cubes, etc. You just need to churn through routine calculus.
- The step right before the messy calculation tells you all you really to know: it's very unlikely the final answer will have a nice form, since that messy nest of quotient rule applications won't.
- If you really desperately do want to know the explicit final answer, a computer can tell it to you with no margin for error or additional effort: in Mathematica,
![Limit[x^2 D[x^2 D[(1 - x^(n + 2))/(1 - x), {x, 2}], {x, 2}],
x -> 1] // FullSimplify](../../images/b54583c049a506edb107f5e1fa92697b.webp)
- If you decide you want variations on this, e.g. cubes of triangular numbers rather than squares, it's often trivial to modify the argument and have a computer tell you the new answer.