Flux is defined in "absolute terms", which is what creates the problem here.
To illustrate this by example, think of $V$ as a screen and $\mathbf F$ as a constant vector field representing water flowing through the screen $V$ (at a constant rate everywhere). This example shows two problems if we don't divide by $|V|$ in our divergence equation, where our intuition about divergence is that it measures to what extent $\mathbf F$ is a source/sink.

In our example, taking $V$ to be large makes $\oint_V \mathbf F \cdot d\mathbf a$ large, and taking $V$ small makes $\oint_V \mathbf F \cdot d\mathbf a$ small. So this is not by itself a good measure of the extent to which $\mathbf F$ is a source/sink, since we know that $\mathbf F$ is constant. Hence $\oint_V \mathbf F \cdot d\mathbf a$ is only really a measure of whether $\mathbf F$ is a source/sink.
Moreover, if $\mathbf F$ is not constant, it could be letting water flow into or out of the screen at different rates in different places, and hence $\oint_V \mathbf F \cdot d\mathbf a$ only measures whether $\mathbf F$ is a net source/sink over the region $V$. To test whether $\mathbf F$ is a source/sink at a point $\mathbf p$, we would approximate by taking a region $V$ about $\mathbf p$ and computing $\oint_V \mathbf F \cdot d\mathbf a$. Again, as $V$ shrinks down to $\mathbf p$, it has $0$ volume, so the integral goes to $0$, and we don't get a very useful measure. So we should instead consider the value of the integral relative to the size of $V$ as we shrink $|V|$ to $0$, thus getting a limit as is familiar with derivatives.