For computational reasons I must map a $3 \times 3$ symmetric, traceless matrix into $\mathbb{R}^2 \times \mathbb{R}^3$ where the components of $\mathbb{R}^2$ are two of the eigenvalues (the third is easily calculable since it's traceless) and the components of $\mathbb{R}^3$ are the three degrees of freedom associated with the element of $SO(3)$ which diagonalizes it -- I've been using three components of a unit quaternion for numerical stability, since that space is locally isomorphic to $SO(3)$. Additionally, I need to find the derivative of this mapping with respect to the degrees of freedom of the matrix, hence a $5\times 5$ Jacobian.
For cases where all eigenvalues are distinct, this is fine because the mapping is locally differentiable. There is a formula given in this answer which seems to work, as well as papers like the ones here and here (behind paywalls, unfortunately). However, for repeated eigenvalues the question here indicates that the mapping is not smooth. Indeed, the mapping isn't even really defined because one may perform an arbitrary rotation on the diagonalized matrix in the plane defined by the eigenvectors of the repeated eigenvalues and still end up with a diagonal matrix.
Given all this, is it possible to add some constraint which makes the mapping (locally) unique and differentiable? One idea is to define the mapping so as to minimize the angle between the original basis and the diagonalized basis. This seems easy enough to compute, but it's unclear to me whether it's unique, and also I don't know how to calculate the derivatives of the eigenvalues/vectors. The answer here suggests using the Plucker embedding, but I'm not really sure what that gets us because it's a map of subspaces, not necessarily bases. Any help is appreciated.
