Where does $\ln x$ come from here?

Question

There's a simple formula for the antiderivative of any function of the form $x^n$; it's $\frac{x^{n+1}}{n+1}$. But in the case of $n=-1$, and only in that case, you get $\ln x$ instead. Even if you go really close to $-1$, for instance $n=-0.9999$, you still get $\frac{x^{0.0001}}{0.0001}$, not anything logarithm-y.

It makes sense that the formula would break, because the formula would then spit out $\frac{x^0}{0}=\frac{1}{0}$, and of course you can't divide by zero. But why specifically $\ln x$?

I understand that it can be derived from the standard Riemann sum definition of an integral; that's not what I'm looking for, I could do that myself. I want an intuitive understanding of why you would get anything well-defined at all here where the formula spits out a division by zero, let alone why it would be specifically a nice function like $\ln x$ as opposed to some horrible mess.

Answer 1

Adayah's suggestion to look at the definite rather than indefinite integral is the right way to think about this but the most interesting step, in my view, is rushed through, which is the identity

$$\lim_{h \to 0} \frac{x^h - 1}{h} = \ln x$$

so let's go over it carefully, because there's something important to understand here and being too cavalier about evaluating this limit skips over the actual enjoyment of understanding it. Why is it that power functions converge to the logarithm as $h \to 0$? You say:

Even if you go really close to $-1$, for instance $n=-0.9999$, you still get $\frac{x^{0.0001}}{0.0001}$, not anything logarithm-y.

But how sure are you really that $\frac{x^{0.0001}}{0.0001}$ isn't "logarithm-y"? Here's a WolframAlpha plot of $\frac{x^h - 1}{h}$ for $h = 0.1, 0.01$ compared to a plot of $\ln x$, on the interval $[1, e]$:

I didn't include $h = 0.001$, let alone $h = 0.0001$, because the graphs are visually indistinguishable! To play around with this more, here's an interactive Desmos plot where you can adjust the value of $n$ in the graph of $\frac{x^{\frac{1}{n}} - 1}{\frac{1}{n}}$ and compare it to the graph of $\ln x$. To my eye you can barely tell the difference for $x \in [0, 10]$ by about $n \approx 50$.

So, what's really going on here? Let's go all the way back to the definition: if $n$ is a natural number for simplicity, and $x$ is a positive real, then $x^{\frac{1}{n}} = \sqrt[n]{x}$ is the positive real which satisfies $(x^{\frac{1}{n}})^n = x$. Now, as $n$ gets very large the function $x^n$ grows very quickly which means that $x^{\frac{1}{n}}$ correspondingly grows very slowly. How slowly?

To be really concrete, let's try to estimate $2^{\frac{1}{1000}}$. Since $1000$ is a very big number this has to be very small, or more precisely very close to $1$. So let's write it as $1 + \varepsilon$ for some $\varepsilon > 0$. We want $(1 + \varepsilon)^{1000} = 2$. Now, binomial expansion gives

$$(1 + \varepsilon)^n \ge 1 + n \varepsilon$$

which means that in order to have $(1 + \varepsilon)^{1000} = 2 \ge 1 + 1000 \varepsilon$ we need $\varepsilon \le \frac{1}{1000}$, so we learn that

$$2^{\frac{1}{1000}} \le 1 + \frac{1}{1000}.$$

More generally, the same argument gives that if $x \ge 1$ then

$$x^{\frac{1}{n}} \le 1 + \frac{x-1}{n}$$

so we've learned that when $n$ gets really big $x^{\frac{1}{n}}$ looks something like $1$ plus a term linear in $\frac{1}{n}$ (formally, what we're seeing here is an estimate of the first two terms of the Taylor series expansion of $x^h$ as $h \to 0^{+}$). But what is that term linear in $\frac{1}{n}$ exactly? If we write

$$x^{\frac{1}{n}} \approx 1 + \frac{f(x)}{n}$$

then exponentiating both sides gives

$$x \approx \left( 1 + \frac{f(x)}{n} \right)^n.$$

Now where have we seen that before? That expression on the RHS famously appears in one of the limits defining the natural exponential: for large $n$ we have $(1 + \frac{x}{n})^n \approx e^x$, so we get

$$\boxed{ x \approx e^{f(x)} }.$$

This gives

$$\boxed{ f(x) \approx \frac{x^{\frac{1}{n}} - 1}{\frac{1}{n}} \approx \ln x }.$$

This isn't a formal argument but it can be turned into one without much difficulty (write $x^h = \exp(h \ln x)$ then take a Taylor series expansion); much more importantly, this line of reasoning doesn't require you to know in advance what logarithms are, just the much simpler concept of taking $n^{th}$ roots. In fact you can use this line of reasoning to discover logarithms (and the natural exponential, simultaneously). This is actually related to the historical pattern of discovery: roughly, Napier calculates logarithms by doing calculations similar to these with $n = 10^7$.

As a bonus, here's how to see that for $h$ small the function $\frac{x^h - 1}{h}$ approximately satisfies the fundamental logarithm law $\ln xy = \ln x + \ln y$. If we write $f(x) = \frac{x^h - 1}{h}$, as above we have $x^h \approx 1 + h f(x)$ as the first-order approximation to the exponential for $h$ small. We then have, on the one hand,

$$(xy)^h \approx 1 + h f(xy)$$

and on the other hand,

$$(xy)^h = x^h y^h \approx (1 + h f(x))(1 + h f(y))$$

and expanding this out and discarding the $h^2$ term gives $f(xy) \approx f(x) + f(y)$ exactly as desired! This approximation is really quite good: you can ask WolframAlpha to verify, for example, that for $h = 0.001$ we have

$$\frac{2^h - 1}{h} = 0.693 \dots$$ $$\frac{3^h - 1}{h} = 1.099 \dots $$ $$\frac{6^h - 1}{h} = 1.793 \dots $$

Answer 2

You can think of it that way: for any $\alpha \neq -1$ an antiderivative $F_{\alpha}(x)$ of $f_{\alpha}(x) = x^{\alpha}$ can be found using Riemann integration:

$$F_{\alpha}(x) = \int \limits_1^x f_{\alpha}(t) \, \text{d} t = \int \limits_1^x t^{\alpha} \, \text{d} t = \left[ \frac{t^{\alpha+1}}{\alpha+1} \right]_1^x = \frac{x^{\alpha+1}-1}{\alpha+1}.$$

This is of course the usual antiderivative $\frac{x^{\alpha+1}}{\alpha+1}$ shifted by a constant, so that $F_{\alpha}(1) = 0$ for all $\alpha \neq -1$. Thanks to this shift, there exists a finite limit:

$$F(x) = \lim_{\alpha \to -1} F_{\alpha}(x) = \lim_{\alpha \to -1} \frac{x^{\alpha+1}-1}{\alpha+1} = \lim_{h \to 0} \frac{x^h-1}{h} = \ln x.$$

On the other hand, since $f_{\alpha}(t) \rightrightarrows \frac{1}{t}$ on every bounded closed interval contained in $(0, \infty)$ as $\alpha \to -1$, we have that

$$\lim_{\alpha \to -1} F_{\alpha}(x) = \int \limits_1^x \frac{1}{t} \, \text{d} t,$$

which means that $\ln x$ is an antiderivative of $\frac{1}{x}$.