19

I have a positive definite matrix $A$. I am going to choose its Frobenius norm $\|A\|_F^2$ as a cost function and then minimize $\|A\|_F^2$. But I think I need to find a reason to convince people it is reasonable to choose $\|A\|_F^2$ as a cost function. So I'm wondering if there are some geometric meanings of the Frobenius norm. Thanks.

Edit: here $A$ is a 3 by 3 matrix. In the problem I'm working on, people usually choose $\det A$ as a cost function since $\det A$ has an obvious geometric interpretation: the volume of the parallelepiped determined by $A$. Now I want to choose $\|A\|_F^2$ as a cost function because of the good properties of $\|A\|_F^2$. That's why I am interested in the geometric meaning of $\|A\|_F^2$.

Shiyu
  • 5,228
  • 3
    $|A|_F^2$ is equal to summation of the squre of all singular values of $A$. Perhaps you could start from the geometric meaning of singular values of a matrix. – Sunni Jul 28 '11 at 01:34
  • Since all matrix norms are equivaluent, it doesn't matter which norm you choose as a cost function. – Sunni Jul 28 '11 at 01:35
  • 1
    @Sunni: For the problem I'm working on, people usually choose $\det A$ as the cost function as $\det A$ has an obvious geometric meaning: the volume of the parallelepiped determined by $A$. I know $|A|_F^2=trace (AA^T)=\sum\sigma_i^2$. But what is the geometric meaning of the sum of singular values? – Shiyu Jul 28 '11 at 02:03

2 Answers2

35

To give intuition, lets look at a particularly nice class of matrices. Suppose the matrix $A$ is symmetric. Then we can find a basis of $n$ orthonormal eigenvectors $v_1,\cdots v_n$ with eigenvalues $\lambda_1,\dots,\lambda_n$. Each eigenvalue geometrically represents the "stretching factor" in the direction of its associated eigenvector.

The Frobenius norm of $A$ in this case is $$\|A\|_F =\sqrt{\text{Tr}(AA^t)}=\sqrt{\text{Tr}(A^2)}=\sqrt{\lambda_1^2+\cdots+\lambda_n^2}.$$ Working in the basis $v_1,\cdots v_n$, consider the box which is the image of the unit cube under $A$. The Frobenius norm is the diagonal of that box, and the determinant is the volume.

The usual norm defined as $\sup_{\|x\|=1}\|Ax\|$ corresponds to the longest side of the box.

Eric Naslund
  • 72,099
  • Very interesting. If not working in the basis $v_1,...,v_n$, $A$ is corresponding a parallelepiped. Then $\det A$ is still the volume of the parallelepiped, can we say $|A|_F$ is still the diagonal of the parallelepiped? – Shiyu Jul 28 '11 at 02:24
  • @Shiyu: By SVD, yes, but it will be the diagonal of a parallelepiped that's of smaller dimension than the ambient space. – anon Jul 28 '11 at 03:27
  • @Eric this is quite nice explanation, however I am not sure what you mean by "consider the box which is the image of the unit cube under A"... What? :-) I think I know what you mean... let us say you have a skewed box, so the Fro-norm is the longest diagonal within the box that touches the corners, while the determinant is the volume of this box? – Spacey Aug 30 '12 at 15:25
  • @Mohammad: Well it is more like an $n$-dimensional parallelogram than a box if that helps. – Eric Naslund Aug 30 '12 at 16:57
  • @EricNaslund is there any meaning if A isn't symmetric? – Maverick Meerkat Mar 12 '20 at 13:44
  • never-mind - if A isn't symmetric eigenvalues should be replaced with singular values. – Maverick Meerkat Mar 13 '20 at 09:54
1

In three dimensions (easier to visualize) we know that the scalar triple product of three vectors, say $a, b, c$, is the determinant of a matrix with those vectors as columns and the modulus is the volume of the parallelepiped spanned by $a, b$ and $c$.

The squared Frobenius norm is the average squared length of the four space diagonals of the parallelepiped. This can easily be shown. The diagonals are:

$d_1 = a + b + c\\ d_2 = a + b - c\\ d_3 = b + c - a\\ d_4 = c + a - b.$

Calculate and sum their squared lengths as $d_1^T d_1 + d_2^T d_2 + d_3^T d_3 + d_4^T d_4.$ Things cancel nicely and one is left with $ 4 ( a^T a + b^T b + c^T c)$ which is exactly four times the square of the Frobenius norm.

The proof in more dimensions is along the same lines, just more sides and diagonals.

The squared Frobenius norm of the Jacobian of a mapping from $\mathbb{R}^m$ to $\mathbb{R}^n$ is used, when it is desired that reductions in volume under the mapping shall be favoured in a minimization task. Because of its form, it is much easier to differentiate the squared Frobenius norm, than any other measure which quantifies the volume change, such as the modulus of the determinant of the Jacobian (which can only be used if $m=n$).

Anita Faul
  • 11