You meant $$\sum_{a,b, ab\le n, a \le k}1 = \sum_{a \le k} \left\lfloor \frac{n}{a} \right\rfloor$$
Then yes it can be computed in $O(\sqrt{n})$ operations because $$\left\{\left\lfloor \frac{n}{a} \right\rfloor, a \ge 1\right\}$$
contains at most $2 \sqrt{n}+1$ elements and
$\left\lfloor \frac{n}{a} \right\rfloor = c \iff \frac{n}{a} = c+r, r \in [0,1)$
$\iff n= ca+ra, a =\frac{n}{c+r}$
$\iff a \in (\frac{n}{c+1},\frac{n}{c}]$ so $\left\lfloor \frac{n}{c}\right\rfloor-\left\lfloor \frac{n}{c+1}\right\rfloor$ many choices for $a$
This way we obtain $$ \sum_{a \le k} \left\lfloor \frac{n}{a} \right\rfloor = \sum_{a =1}^{ \min(\lfloor \sqrt{n}\rfloor,k)} \left\lfloor \frac{n}{a} \right\rfloor+ \sum_{c=\lfloor n/k\rfloor}^{\lfloor n/\lfloor \sqrt{n}\rfloor\rfloor-1} \min(k,\left\lfloor \frac{n}{c}\right\rfloor)-\min(k,\left\lfloor \frac{n}{c+1}\right\rfloor)$$