I was looking at The accuracy from left to right and that from right to left of the floating point arithmetic sums which was asking for the accuracy of $$\sum_{k=1}^n\frac{1}{k^2}$$ from right to left, and I was wondering how summing from different directions gives you different accuracy?
Edit: I am specifically looking for where and how that $O(\log n)$ comes from in the link.
When summing positive numbers is is critical that you sum the smallest term first in order to minimize the worst case behaviour of the round off error. This is most easily understood by studying an extreme example. Specifically, compute \begin{equation} 2 = 1 + \underset{2^{24} \text{copies}}{\underbrace{u + u + \dotsc u}}, \quad + u = 2^{-24} \end{equation} using single precision floating point numbers. In the default rounding mode the floating point representation of $1 + 2^{-24}$ is $1$. It follows, that if we sum the largest terms first, then the computed sum is $1$, rather than $2$. However, if we sum the smallest terms first, then the computed sum is $2$.
In general, if we sum $n$ floating point numbers $x_i$ together, using the simple algorithm \begin{equation} s_1 = x_1, \quad s_i = s_{i-1} + x_i \end{equation} the computed sum satisfies \begin{multline} \hat{s}_n = x_1 (1 + \theta_1^{(n-1)}) + x_2( 1 + \theta_2^{(n-1)}) + x_3 (1 + \theta_3^{(n-2)}) \\ + x_4 (1 + \theta_4^{(n-3)}) + \dotsc + x_{n-1} (1 + \theta_{n-1}^{(2)}) + x_n(1 + \theta_n^{(1)}) \end{multline} where \begin{equation} |\theta_j^{(k)}| \leq \gamma_k = \frac{ku}{1 - ku}. \end{equation} From this representation of the computed sum we see that if a large number enters the computation early, then it has a large change of making a large contribution to the error. This explains why we should sum the smallest terms first.
I have quietly assumed that the calculation of the sum does not overflow.
Turning to the toward the special case of $s = \sum_{k=1}^n \frac{1}{k^2}$ we choose to sum the terms in increasing order, so that \begin{equation} x_k = \frac{1}{(n-k+1)^2}. \end{equation} In general the error can be written as \begin{equation} \hat{s} - s = \sum_{k=1}^n x_k \theta_k^{(n_k)} \end{equation} where $|\theta_k^{(l)}| \leq \gamma_l$ and $n_k = \min\{ n - 1, n - k + 1\} \leq n - k + 1$. In our special case, the expression for the error is \begin{equation} \hat{s} - s = \sum_{k=1}^n \frac{1}{(n-k+1)^2} \theta_k^{(n_k)} \end{equation} which allows us to estimate \begin{equation} |\hat{s} - s| = \sum_{k=1}^n \frac{1}{(n-k+1)^2} \left|\theta_k^{(n_k)}\right| \leq \sum_{k=1}^n \frac{\gamma_{n-k+1}}{(n-k+1)^2}. \end{equation} In order to continue we exploit $\gamma_l \approx l u$ which is a good approximation as long as $lu \ll 1$. It follows that \begin{equation} |\hat{s} - s| \lesssim \sum_{k=1}^n \frac{(n-k+1)u }{(n-k+1)^2} = u \sum_{k=1}^n \frac{1}{(n-k+1)} = u \sum_{k=1}^n \frac{1}{k} \approx u \log(n). \end{equation} The last approximation is obtained by regarding the sum \begin{equation} \sum_{k=1}^n \frac{1}{k} \end{equation} as a crude approximation of the integral \begin{equation} \int_1^n \frac{1}{t}dt = \log(n). \end{equation} This completes the treatment of the special case.
