The gradient as a row versus column vector

Question

Kaplan's Advanced Calculus defines the gradient of a function $f : \mathbb{R^n} \to \mathbb{R}$ as the $1 \times n$ row vector whose entries respectively contain the $n$ partial derivatives of $f$. By this definition then, the gradient is just the Jacobian matrix of the transformation.

We also know that using the Riesz representation theorem, assuming $f$ is differentiable at the point $x$, we can define the gradient as the unique vector $\nabla f$ such that

$$ df(x)(h) = \langle h, \nabla f(x) \rangle, \quad h \in \mathbb{R}^n $$

Assuming we ignore the distinction between row vectors and column vectors, the former definition follows easily from the latter. But, row vectors and column vectors are not the same things. So, I have the following questions:

1. Is the distinction here between row/column vectors important?
2. If (1) is true, then how can we know from the second defintion that the vector in question is a row vector and not a column vector?

Answer 1

Yes, the distinction between row vectors and column vectors is important. On an arbitrary smooth manifold $M$, the derivative of a function $f : M \to \mathbb{R}$ at a point $p$ is a linear transformation $df_p : T_p(M) \to \mathbb{R}$; in other words, it's a cotangent vector. In general the tangent space $T_p(M)$ does not come equipped with an inner product (this is an extra structure: see Riemannian manifold), so in general we cannot identify tangent vectors and cotangent vectors.

So on a general manifold one must distinguish between vector fields (families of tangent vectors) and differential $1$-forms (families of cotangent vectors). While $df$ is a differential form and exists for all $M$, $\nabla f$ can't be sensibly defined unless $M$ has a Riemannian metric, and then it's a vector field (and the identification between differential forms and vector fields now depends on the metric).

If one thinks of tangent vectors as column vectors, then $\nabla f$ ought to be a column vector, but the linear functional $\langle -, \nabla f \rangle$ ought to be a row vector. A major problem with working entirely in bases is that distinctions like these are frequently glossed over, and then when they become important students are very confused.

---

Some remarks about non-canonicity. The tangent space $T_p(V)$ to a vector space at any point can be canonically identified with $V$, so for vector spaces we don't run into quite the same problems. If $V$ is an inner product space, then in the same way it automatically inherits the structure of a Riemannian manifold by the above identification. Finally, when people write $V = \mathbb{R}^n$ they frequently intend $\mathbb{R}^n$ to have the standard inner product with respect to the standard basis, and this equips $V$ with the structure of a Riemannian manifold.

Answer 2

Here's a simple heuristic. TL;DR: If the domain of $f: R^n\to R$ is a space of column vectors, then $f'(x)$ needs to be a row vector for the linear approximation property to make sense.

To make the distinction between row and column vectors explicit, I'll write $R^{k\times 1}$ and $R^{1\times k}$ for the spaces of $k$-dimensional column and row vectors of real numbers, respectively.

If $f:R^{n\times 1}\to R^{m\times 1}$, then, for every $x\in R^{n\times 1}$, the derivative $f'(x)$ is characterized by its linear approximation property, $$ f(x + h) \approx f(x) + f'(x)h, $$ for small $h\in R^{n\times 1}$. Now think about the sizes of the matrices involved: $f(x)$ has size $m\times 1$, so $f'(x)h$ also needs to have size $m\times 1$ for the right hand side to make sense. But $h\in R^{n\times 1}$, so $f'(x)h$ has size $m\times 1$ exactly when $f'(x)$ has size $m\times n$.

In particular, if $f:R^{n\times 1}\to R^{1\times 1}$, then $f'(x)$ needs to be an $1\times n$ matrix, i.e., a row vector.