1

If $A$ is a $n\times m$ matrix and $B$ a $m \times k$ matrix (both real), can then something be said about how the singular value of $A$ and $B$ are related to the singular values of the product $AB$?

I'd be interested ideally in a formula that gives the singular values of $AB$ in terms of the singular values of $A$ and $B$; or, if that is not possible, at least in an upper bound of the singular values of $AB$ in terms of the singular values of $A$ and $B$.

I have spent some time googling know and all I could find where research articles describing the singular value decomposition of $AB$, but extracting what this means for the singular values is actually non-trivial because a ton of notation is introduced and the theorems are straightforward to read, so I was wondering whether perhaps more "readable" reference or answer could be given here? I'm guessing since I'm only after the singular values, perhaps there is a more straightforward answer?

temo
  • 5,237
  • I'm also really interested in this question! ^_^

    One application of this will be assuming a composition of functions (e.g. neural net) $f\circ g \circ h$ then the jacobian of the total function will be a matrix multiplication of the jacobian of each $df \cdot dg \cdot dh$.

    Then it will be really useful if something can be said about the total jacobian's conditional number from the conditional number of individual jacobian

     – Binxu Wang 王彬旭 May 17 '22 at 01:39
  • 1
    one can prove that the singular values of $(AB)$ are weakly majorized by those of $A$ plus those of $B$ i.e. $\Sigma_{AB}\preceq_w \Sigma_A+\Sigma_B$ where $\Sigma_X$ is a diagonal matrix with the singular values of $X$ sorted from largest to smallest. I don't know of an easy proof though Olkin and Marshall's book on majorization should have it. As for "I'd be interested ideally in a formula that gives the singular values of in terms of the singular values of and " -- why don't you consider examples with (square) diagonal $A,B$ and confirm to yourself that this is impossible. – user8675309 May 17 '22 at 04:46
  • sorry, that should have read $\Sigma_{AB}\preceq_w \Sigma_{A}\Sigma_{B} $ – user8675309 May 17 '22 at 07:19
  • @user8675309 First, what does weak majorization mean? (I know of the word "weak" only in functional analytic contexts, when we speak, e.g. of weak convergence.) Second, I don't quite understand what you mean that a formula is impossible? For the example you have given it seems perfectly possible to get a formula (a back-of-the-envelope computation suggests the singular values of the product is in this case the product oft the singular values, which in turn are the absolute values of the elements on the diagonal) ... [...] – temo May 17 '22 at 10:59
  • [...] Although another related answer here https://math.stackexchange.com/questions/2448088/singular-values-of-ab-and-ba-where-a-and-b-are-rectangular-matrices?rq=1 actually seems to show for a more involved example that there does not need to be a relationship between the singular values. (The answer itself doesn't show that, but you can use the matrices that appear there to see that the singular values of $AB$ do not depend in any way on the singular values of $A$, as they don't depend on the $x$ entry from that matrix! Though the singular values of $AB$ depend on the $y$ entry from $B$.) – temo May 17 '22 at 10:59
  • I had a look in the book, thanks for pointing that out. The definition of "weak" seems to be rather technical; are you sure thought that was it is that book applies? It seems to be that mostly the case of square matrices is treated. (If you can give me a direct reference for your statement, I'd be happy with that) – temo May 17 '22 at 11:14
  • "It seems to be that mostly the case of square matrices is treated." You should be able to confirm that you can always zero pad non-square matrices $A,B$ to make them square without altering non-zero singular values, then use results about square matrices. I also don't see that you've played around with diagonal matrices to confirm that a formula mapping from singular values of A,B to A and B doesn't make much sense. – user8675309 May 17 '22 at 16:45
  • 1
    @BinxuWang王彬旭 My use case is actually also machine learning, though a different application. ;) And I'm also at an Ivy League uni haha. Though I have the feeling the conditional number of the jacobian of a NN has been studied before :/ – temo May 19 '22 at 13:28

0 Answers0