Let $f$ be the function defined for a positive integer $N$ as follows:
$$
f(N)
=
\sum_{1\le n\le N}\{n\sqrt 2\}\ .
$$
We need a control of $f(N)-N/2$.
Here, i try to give arguments for the idea extracted from the following experimental computation:
Experiment:
We approximate $\sqrt 2$ by using continued fractions, for such a fixed approximation,
$P/Q$ say, we compute $f(Q)$ and $f(Q)-Q/2$. For the first few values of $Q$...
def f(N):
return sum( [ RR(k*sqrt(2)).frac() for k in xrange(1, N+1) ] )
cf = continued_fraction( sqrt(2) )
# consider the first 15 "convergents" P/Q and compute f(Q) - Q/2 for them
for frac in cf.convergents()[:15]:
P, Q = frac.numerator(), frac.denominator()
print "sqrt(2) ~ %s :: Q = %s :: f(Q)-Q/2 ~ %s" % ( frac, Q, f(Q)-Q/2 )
Results:
sqrt(2) ~ 1 :: Q = 1 :: f(Q)-Q/2 ~ -0.0857864376269049
sqrt(2) ~ 3/2 :: Q = 2 :: f(Q)-Q/2 ~ 0.242640687119285
sqrt(2) ~ 7/5 :: Q = 5 :: f(Q)-Q/2 ~ -0.286796564403573
sqrt(2) ~ 17/12 :: Q = 12 :: f(Q)-Q/2 ~ 0.308657865101423
sqrt(2) ~ 41/29 :: Q = 29 :: f(Q)-Q/2 ~ -0.317100367703610
sqrt(2) ~ 99/70 :: Q = 70 :: f(Q)-Q/2 ~ 0.320702497141440
sqrt(2) ~ 239/169 :: Q = 169 :: f(Q)-Q/2 ~ -0.322176510488248
sqrt(2) ~ 577/408 :: Q = 408 :: f(Q)-Q/2 ~ 0.322790161566445
sqrt(2) ~ 1393/985 :: Q = 985 :: f(Q)-Q/2 ~ -0.323043813132131
sqrt(2) ~ 3363/2378 :: Q = 2378 :: f(Q)-Q/2 ~ 0.323148970494003
sqrt(2) ~ 8119/5741 :: Q = 5741 :: f(Q)-Q/2 ~ -0.323192510470562
sqrt(2) ~ 19601/13860 :: Q = 13860 :: f(Q)-Q/2 ~ 0.323210559656218
sqrt(2) ~ 47321/33461 :: Q = 33461 :: f(Q)-Q/2 ~ -0.323217967510573
sqrt(2) ~ 114243/80782 :: Q = 80782 :: f(Q)-Q/2 ~ 0.323221431812271
sqrt(2) ~ 275807/195025 :: Q = 195025 :: f(Q)-Q/2 ~ -0.323220559774200
So the difference is in these cases "surprisingly" in the interval $[-1,1]$.
The estimation:
Let us try to convert these observation into a proof.
Let $a$ be $\sqrt 2-1$, so $a\in(0,1/2)$.
(I need $a<1$ below.) We have $\{n\sqrt 2\}=\{na\}$.
Let $P/Q$ be a rational approximation of $a$, an irreducible fraction, namely one provided by the
continued fraction convergents of $a$, which form an "alternated sequence around $a$" $\dots P/Q, P'/Q', P''/Q'',\dots$,
and the difference between two consecutive convergents $P/Q$ and $P'/Q'$ is $\pm 1/(QQ')$.
In particular, $Q,Q'$ are relatively prime.
We will also use in the following the "next convergent" $P'/Q'$.
Fix some natural $n$ with $0<n<Q$. So $nP/Q<n$ is not an integer. (If $Q|(nP)$, then $Q|n$.)
Assume now there is an integer between $na$ and $nP/Q$.
Then the same integer is in the bigger interval between $nP'/Q'$ and $nP/Q$, which is of length
$1/(QQ')<1/Q$. But from $nPQ\not\in\Bbb N$ to the next integer is a distance of at least $1/Q$.
Contradiction.
Our assumption is false.
So $na$ and $nP/Q$ have the same floor.
Then
$$
\left\{na\right\}
=
\left\{n\frac PQ+n\left(a-\frac PQ\right)\right\}
$$
lies between the numbers
$$
\left\{n\frac PQ\right\}\pm \underbrace{\left\{n\left(a-\frac PQ\right)\right\}}_{<n/(QQ')}\ .
$$
Now let $n$ run in a set $S=S(Q)$ of $Q$ consecutive elements. Then
$$
\begin{aligned}
\sum_{\substack{n\in S(Q)\\Q\not|n}} \{na\}
\text{ lies between }
&\sum_{\substack{n\in S(Q)\\Q\not|n}} \left\{n\frac PQ\right\}
\pm \sum_{\substack{n\in S(Q)\\Q\not|n}}\left\{n\left(a-\frac PQ\right)\right\}
\\
\text{ thus between }
&\sum_{0<n<Q} \frac nQ
\pm \frac 1{QQ'}\sum_{n\in S(Q)}{\color{red}{n}}
\\
=
&\frac 12(Q-1)
\pm \frac 1{QQ'}\sum_{n\in S(Q)}{\color{red}{n}}
\ .
\end{aligned}
$$
(Later edit in red.)
This can now be converted to a proof for the stated result as follows.
I try to explain my feeling of a procedure using an example, $N=100000$ say.
Recall the following denominators "$Q$" of the convergents of $a=\sqrt 2-1$:
$1$, $2$, $5$, $12$, $29$, $70$, $169$, $408$, $985$, $2378$, $5741$, $13860$, $33461$, $80782$, $195025$.
Using $Q=80782$ and $S(Q)=(N-Q,N]\cap\Bbb Z$ we obtain a deviation from $\frac 12|S(Q)|$ which is less than (one plus)
$\frac {N(N+1)/2}{QQ'}<\frac{Q'Q'/2}{QQ'}<2$. (Here, $Q'/Q\approx \sqrt 2+1$, which is the limit of the ration of two
consecutive convergents.)
Then for the "new N", which is $N-Q=19281$ we use the "new Q" $13860$. And so on.
Using this we get a deviation of $f(N)$ from $N/2$ which is of the shape $2\log_{\sqrt 2+1} N$.
(Have to submit, hope that the idea is clear, this was more important for me than to write things rigurous,
and possibly deflect from the idea.)