Prove that if $A^TB$ and $AB^T$ are symmetric, then there exist orthogonal $U,V$ such that $UAV^T$ and $UBV^T$ are diagonal

Question

The following fact is used as a step in this paper, but is not proved.

Proposition: Given $A,B \in \Bbb R^{m \times n}$, there exist orthogonal matrices $U,V$ (of sizes $m$ and $n$) such that $UAV^T$ and $UBV^T$ are diagonal if and only if the matrices $A^TB$ and $AB^T$ are both symmetric.

I would like to prove this statement. Any ideas here would be appreciated!

---

Thoughts on the problem:

The "only if" direction is easy to check by direct computation.

For the "if" direction, it is notable that $A^TB$ and $AB^T$ is symmetric if and only if the matrices $\hat A, \hat B$ commute, where $$ \hat A = \pmatrix{0&A\\A^T&0}, \quad \hat B = \pmatrix{0&B\\B^T&0}. $$ I had hoped that it would be straightforward to prove the desired statement using the fact that $\hat A,\hat B$ are simultaneously diagonalizable: the statement that we are trying to show amounts to saying that the matrices $\hat U \hat A \hat U^T, \hat U \hat B \hat U^T$ are both diagonal, with $$ \hat U = \pmatrix{U & 0\\0 & V} $$ for orthogonal matrices $U,V$. While I can guarantee that a diagonalizing orthogonal matrix $\hat U$ exists, I do not see how to guarantee that such a $\hat U$ exists of the appropriate block-diagonal form.

---

With an SVD, we can apply this approach to the reduced case where $A = \Sigma$ is diagonal with non-negative entries. In the case where $m = n$, this looks like $$ \hat \Sigma = \pmatrix{0 & \Sigma \\ \Sigma & 0}, \quad \hat B = \pmatrix{0 & B\\B^T & 0}. $$ Applying the change of basis $\hat U_0$ given by $$ \hat U_0 = \frac 1{\sqrt{2}}\pmatrix{I&I\\I&-I} $$ to both matrices gives us the block matrices $$ \pmatrix{\Sigma & 0\\0 & -\Sigma}, \quad \frac 12 \pmatrix{B + B^T & B^T - B\\ B - B^T & -(B + B^T)}. $$ Perhaps it is easier to consider the orthogonal matrices that simultaneously diagonalize this pair.

Answer 1

Some simple blockwise operations will suffice. Let $$ A=\pmatrix{S&0&0\\ 0&0&0}\ \text{ and }\ B^T=\pmatrix{X&E\\ Y&F\\ Z&G} $$ where $S$ is a positive diagonal square matrix. If $$ AB^T=\pmatrix{SX&SE\\ 0&0}\ \text{ and }\ (A^TB)^T=B^TA=\pmatrix{XS&0&0\\ YS&0&0\\ ZS&0&0} $$ are symmetric, then both $SX$ and $XS$ are symmetric and $E,Y,Z$ are zero matrices. Since $S$ is symmetric and nonsingular, that $SX$ and $XS$ are symmetric means that $X$ is a symmetric matrix that commutes with $S$. Therefore $S$ and $X$ are simultaneously orthogonally diagonalisable. Now $$ A=\pmatrix{S&0&0\\ 0&0&0}\ \text{ and }\ B^T=\pmatrix{X&0\\ 0&F\\ 0&G}. $$ Since $S$ and $X$ can be simultaneously diagonalised by orthogonal similarity and $\pmatrix{F\\ G}$ can be diagonalised by SVD, we are done.