Consider a function $f : [-1,1] \rightarrow \mathbb R$. The Riemann integral of $f$, denoted by $\int_{-1}^1 f dx$ is defined as the limit of the Riemann sums of $f$, if said limit exists. However, $f : \mathbb R\setminus\{0\} \rightarrow \mathbb R$ has a non-removable singularity at $0$, and to be more precise, an unbounded singularity, which intuitively, means that $f$ "blows up" at $0$. Thus, even if we let $f(0) = a$ for some real $a$, the Riemann sums would not converge, due to this singularity. Since the Riemann sums do not converge, $\int_{-1}^1 f dx$ simply does not exist in this case.
I understand the general concept of there being 'different' infinities because functions have different rates of change. However, I find this one especially baffling because this function fits the criteria... I know that all odd functions fit this criteria, but I'm writing this explicitly because this means that they shouldn't be different infinities at all; they start at opposite points, and go at the same rates at all times.
This is inaccurate. I would refrain from trying to use the concept of infinity to intuitively understand this. Also, "rate" in this context does not refer to the derivative. As I just explained, $\int_{-1}^1 f dx$ does not exist. So instead, it makes sense to talk about $\int_{-1}^{-\delta}f dx + \int_{\epsilon}^{1} f dx$, and let $\delta \rightarrow 0^+, \epsilon \rightarrow 0^+$. The fact that $\delta$ and $\epsilon$ are independent of each other is what people actually mean when they say "the rates are different". As you will see, this limit with $\delta \rightarrow 0^+, \epsilon \rightarrow 0^+$ does not exist. Hence, the integral in question does not exist, even as an improper integral.
The Cauchy principal value is what you get when you let $\delta = -\epsilon$. Intuition would suggest this due to the symmetry between $-1$ and $1$, but this is actually only the average. What I mean is that, in a very precise sense, if you look at the average value of $\int_{-1}^{-\delta}f dx + \int_{\epsilon}^{1} f dx$ near the singularity, and you then evaluate the limit of the average, you get 0. That is actually the essence of what the Cauchy principal value is: an operator that takes a singularity and replaces it by the average of the function near that singularity, in a precisely well-defined way.
If this integral were expressed as a series representing..., there would always be a pair of points that cancel out, despite both integrals approaching infinity as they get closer to 0. This makes it even more baffling, because I remember the sum of all whole numbers being -1/12 - the derivation of which required the canceling of many pairs of numbers representing the series.
The "sum" of "whole" numbers is not $-1/12$. This is painfully inaccurate, and the Numberphile video on the subject was somewhat misleading, because they did not sufficiently emphasize the technical details behind what they were claiming. For starters, you cannot have a "sum" of infinitely many objects. A "sum" is inherently an operation on n-tuples. You can, however, used sums to define the discrete integral of a sequence, and evaluate this discrete integral, then find what the discrete integral converges to, and in the context of calculus, this is often called an "infinite sum", even though this is just abuse of language. Also, the Ramanujan summation that gives $-1/12$ for an answer when the sequence of positive integers is given is not actually the limit of the discrete integral of this sequence. This is a whole rabbit hole of its own that deserves to be discussed in another post, but the point is, no mathematician is "cancelling out" any "infinities" to obtain any valid results. At no point are we ever even dealing with infinite quantities. This is entirely a matter of limits, and understanding how the limits work intuitively and in context.
Anyway, to summarize: the Riemann integral of $f$ does not exist, because $f$ "blows up" at 0, and the improper integral also does not exist, because $\lim_{\delta\to 0^+, \epsilon\to 0^+} \ln(\frac{\epsilon}{\delta}$ does not exist. There are no "infinities" to subtract here. That is what your intuition is failing to grasp.