Here's a neat method from de Bruijn's book*. $\DeclareMathOperator{rre}{Re}$
Let $z$ be a complex parameter with $\rre z > 0$, $z \neq 1$. By telescoping we have
$$
\sum_{k=1}^{n} k^{-z} = \frac{n^{1-z}}{1-z} + \sum_{k=1}^{n} \left(k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right).
$$
The sum on the right converges as $n \to \infty$ since
$$
k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z} \sim - \frac{z}{2} k^{-1-z}
$$
as $k \to \infty$, so we can write
$$
\sum_{k=1}^{n} k^{-z} = \frac{n^{1-z}}{1-z} + \sum_{k=1}^{\infty} \left(k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right) + \epsilon_n(z), \tag{1}
$$
where $\epsilon_n(z) \to 0$ as $n \to \infty$ for fixed $z$. We'll show at the end of the answer that the sum
$$
S(z) := \sum_{k=1}^{\infty} \left(k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right)
$$
is analytic in the region $\rre z > 0$, $z \neq 1$. Taking $\rre z > 1$ and sending $n \to \infty$ in $(1)$ we find that
$$
S(z) = \sum_{k=1}^{\infty} k^{-z} = \zeta(z),
$$
so by analytic continuation we must also have $S(z) = \zeta(z)$ for all $z$ with $\rre z > 0$, $z \neq 1$. Equation $(1)$ can therefore be written
$$
\sum_{k=1}^{n} k^{-z} = \frac{n^{1-z}}{1-z} + \zeta(z) + \epsilon_n(z). \tag{2}
$$
Taking $z = 1/2$ in $(2)$ yields
$$
\sum_{k=1}^{n} \frac{1}{\sqrt{k}} = 2\sqrt{n} + \zeta(1/2) + \epsilon_n(1/2),
$$
so we may conclude that
$$
\lim_{n \to \infty} 2\sqrt{n} - \sum_{k=1}^{n} \frac{1}{\sqrt{k}} = -\zeta(1/2).
$$
Now let's prove that
$$
S(z) = \sum_{k=1}^{\infty} \left(k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right)
$$
is analytic in the region $\rre z > 0$, $z \neq 1$. It suffices to show that the sum converges uniformly with respect to $z$ as long as $z$ remains in a compact subset of $\rre z > 0$, $z \neq 1$. Indeed, sequences of analytic functions which converge uniformly on all compact subsets of a domain converge to an analytic function on that domain.
To start let's write
$$
k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z} = \frac{k^{1-z}}{1-z} \left[\left(1-\frac{1}{k}\right)^{1-z} - 1 + \frac{1-z}{k}\right]. \tag{3}
$$
The term in brackets is none other than the function $f(w) := (1-w)^{1-z}$ minus the first two terms of its Maclaruin series and evaluated at $w = 1/k$. We therefore expect a bound like
$$
\left|f(w) - \sum_{j=0}^{1} \frac{f^{(j)}(0)}{j!}w^j\right| \leq Cw^2 = \frac{C}{k^2},
$$
and we just need to show that $C$ can be chosen independently of $z$.
If the Maclaurin series for a function $f$ has radius of convergence $R > 0$ then (after some work) it follows from Cauchy's integral theorem that
$$
f(w) - \sum_{j=0}^{N} \frac{f^{(j)}(0)}{j!} w^j = \frac{w^{N+1}}{2\pi i} \int_C \frac{f(\zeta)}{\zeta^{N+1}(\zeta - w)}\,d\zeta,
$$
where $C$ is the circle $|\zeta| = r < R$ and $|w| < r$. So, taking $f(w) = (1-w)^{1-z}$, $N = 1$, $w = 1/k$, and $r = 1/2$ we have, for $k \geq 3$,
$$
\left(1-\frac{1}{k}\right)^{1-z} - 1 + \frac{1-z}{k} = \frac{1}{2\pi i k^2} \int_{|\zeta| = 1/2} \frac{(1-\zeta)^{1-z}}{\zeta^2 (\zeta - 1/k)}\,d\zeta.
$$
Now assume $\rre z \geq \delta$ and $|1-z| \geq \delta$ for some $0 < \delta < 1$. By taking the modulus of the above equation we get
$$
\begin{align}
\left|\left(1-\frac{1}{k}\right)^{1-z} - 1 + \frac{1-z}{k}\right| &= \left|\frac{1}{2\pi i k^2} \int_{|\zeta| = 1/2} \frac{(1-\zeta)^{1-z}}{\zeta^2 (\zeta - 1/k)}\,d\zeta\right| \\
&\leq \frac{1}{2\pi k^2} \int_{|\zeta| = 1/2} \frac{|1-\zeta|^{1-\rre z}}{|\zeta|^2 |\zeta - 1/k|}\,|d\zeta| \\
&\leq \frac{1}{2\pi k^2} \int_{|\zeta| = 1/2} \frac{2}{(1/2)^2 (1/2 - 1/3)}\,|d\zeta| \\
&= \frac{2}{2\pi (1/2)^2 (1/2-1/3)k^2} \int_{|\zeta| = 1/2} |d\zeta| \\
&= \frac{2 \pi}{2\pi (1/2)^2 (1/2-1/3)k^2} \\
&= \frac{24}{k^2}.
\end{align}
$$
Combining this with $(3)$ yields
$$
\begin{align}
\left|k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right| &= \frac{k^{1-\rre z}}{|1-z|} \left|\left(1-\frac{1}{k}\right)^{1-z} - 1 + \frac{1-z}{k}\right| \\
&\leq \frac{k^{1-\delta}}{\delta} \left|\left(1-\frac{1}{k}\right)^{1-z} - 1 + \frac{1-z}{k}\right| \\
&\leq \frac{24}{\delta} k^{-1-\delta}
\end{align}
$$
for $k \geq 3$, which is what we wanted to show. It follows that the series
$$
\sum_{k=1}^{\infty} \left(k^{-z} - \frac{k^{1-z}}{1-z} + \frac{(k-1)^{1-z}}{1-z}\right)
$$
converges uniformly in the region $\{z \in \mathbb C : \rre z \geq \delta \text{ and } |1-z| \geq \delta\}$ for every $\delta > 0$ and hence that $S(z)$ is analytic in the region $\rre z > 0$, $z \neq 1$.
* N. G. de Bruijn, Asymptotic Methods in Analysis, section 3.5.