Proving identity $ \binom{n}{k} = (-1)^k \binom{k-n-1}{k} $. How to interpret factorials and binomial coefficients with negative integers.

Question

I at a loss on how to prove $$ \binom{n}{k} = (-1)^k \binom{k-n-1}{k} \tag{1} $$.

I want to prove this because it appears to be a fundamental result, that is applied liberally in combinatorics problems such as Proving $\sum\limits_{m=0}^M \binom{m+k}{k} = \binom{k+M+1}{k+1}$, especially in this answer.

In this sketch of a pr oof, I start with the RHS of $\large \text{(1)}$, namely $(-1)^k\binom{k-n-1}{k}$.

(I also assume $k \ge 0, n \ge 0, k \le n$, neither of which may be necessary, however, it's easier for me to think about Binomial Coefficients and factorials with positive integer arguments.)

Then, in a series of algebraic steps that start with $(-1)^k\binom{k-n-1}{k}$, I attempt to reach a result that shows these are equal to $\binom{n}{k}$ as follows

$$\begin{align} &(-1)^k\binom{k-n-1}{k} = \frac{(k-n-1)!}{k!(k-n-1-k)!}(-1)^k = (-1)^k\frac{(k-n-1)!}{k!(-n-1)!} \\ \end{align}$$

But, under these assumptions $k \le n, k \ge 0, n \ge 0$, I don't know how to make sense of what $(-1)^k \large \frac{(k-n-1)!}{k!(-n-1)!}$ simplifies to, since it seems to me that, $(k-n-1)!$ is the factorial of a negative integer, which evaluates to ComplexInfinity.

So, my main question is how to begin computing $(-1)^k \large \frac{(k-n-1)!}{k!(-n-1)!}$ when you know that for some values of $x, n$, that $(k-n-1)!$ = ComplexInfinity or $(-n-1)! = $ ComplexInfinity

One approach I tried was to go back to the RHS of $\large (1)$: $\small (-1)^k \large \frac{(k-n-1)!}{k!(-n-1)!}\left(\frac{-1}{-1} \right)$ which almost seems helpful, except I'm not sure that, in general, $(-1)(k-n-1)! = (n+1-k)!$

I rewrote the above with the $\Gamma$ function instead of factorials. Does this approach lead to the correct solution, where I fell short with factorials?

$$\begin{align} &= \frac{\Gamma(k-n)}{\Gamma(k+1)\Gamma(-n)}(-1)^k \end{align}$$

Answer 1

The given formula clearly assumes that binomial coefficients $\binom nk$ are defined at least for all $n\in\Bbb Z$ (but one may stick to $k\in\Bbb N$): the only instance of the formula that avoids negative upper indices is when $0\leq n<k$ but then it gives the entirely uninteresting identity $0=0$. The usual combinatorial interpretation of$~\binom nk$ can of course not be applied when $n<0$, so something else must guide extending the definition to such cases.

It is not a priori clear that one should try to extend the definition at all, but it turns out to be quite useful. Here are some reasons. One observation is that for fixed $k$, the value $\binom nk$ is a degree $k$ polynomial function of $n$, and this suggests that one use this as a definition for cases where $n\notin\Bbb N$. Indeed with $$ \binom Xk \overset {\rm def} = \frac{X(X-1)\ldots(X-k+1)}{k!}\in\Bbb Q[X] \qquad\text{for any $k\in\Bbb N$}\tag1 $$ we get the usual binomial coefficients at $X:=n\in\Bbb N$, and one can define $\binom xk$ for almost any value $x$ by substitution into the polynomial $\binom Xk$. This works notably for negative integers, which is the most important extension case, but it also works for rational or even complex numbers $x$ (algebraically the only restriction is that $x$ must be in a ring containing $\Bbb Q$; this allows much freedom, but excludes most notably elements of $\Bbb Z/n\Bbb Z$). Writing out $\binom{-X}k$ gives $\binom{-X}k=(-1)^k\binom{X+k-1}k$, so the formula of the question indeed becomes a fundamental identity. Another fundamental identity is Pascal's recurrence $$ \binom{X+1}k=\binom X{k-1}+\binom Xk \qquad\text{for integer $k>0$}\tag2 $$ which is easily checked. It should be noted by contrast that symmetry $\binom nk=\binom n{n-k}$ is no longer meaningful or true when $n\in\Bbb N$ is replaced by a general $x$.

The main fact that makes the newly defined values $\binom xk$ useful is the formal power series identity $$ (1+X)^\alpha = \sum_{k\in\Bbb N}\binom\alpha kX^k \tag 3 $$ that holds for them. I should say it is initially only an identity for $\alpha\in\Bbb Z$, when the left hand side is unambiguously defined as formal power series; for $\alpha\geq0$ it is the binomial formula (with $Y=1)$, while for $\alpha<0$ it can be proved by induction on $-\alpha$ using $(2)$. However since the additive law for exponents can also be proved, the RHS of $(3)$ provides the canonical choice for fractional powers of $(1+X)$ as power series, and in fact serves as definition of $(1+X)^\alpha$ for arbitrary $\alpha$.

The idea chosen in the question to write binomial coefficients in terms of factorials and then to replaces these by instances of the Gamma function is rather unfruitful. It leads to definitional complications (because the Gamma function is undefined) exactly for the case of negative integer upper index, which is of most direct interest.

Answer 2

In this answer, I show that $$ \begin{align} \binom{-n}{k} &=\frac{\overbrace{-n(-n-1)(-n-2)\dots(-n-k+1)}^{k\text{ factors}}}{k!}\\ &=(-1)^k\frac{(n+k-1)(n+k-2)(n+k-3)\dots n}{k!}\\ &=(-1)^k\binom{n+k-1}{k} \end{align} $$

Answer 3

When $a$ is not a nonnegative integer, $\binom{a}{k}$ is defined by $$\binom{a}{k} = \frac{a(a-1)\cdots(a-k+1)}{k!},$$ where the numerator has $k$ terms.

Using this definition you should be able to prove your result.