I have an object(cow).
This cow was translated and rotated(with arbitrary axis) in the modeling space for several times.
And I can't know how much cow is translated or rotated.
Only I can obtain is this cow's current matrix and it's inverse.
In this circumstance, how can I rotate the cow around the x-axis in the viewing space?
(center of rotation is at the center of modeling space)
A rotation around the x-axis is achieved by this matrix $$ \mathtt{T}_\mathrm{rot X} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0& \cos \alpha & -\sin \alpha& 0 \\ 0 & \sin \alpha & \cos \alpha & 0 \\ 0 & 0 & 0 &1 \\ \end{bmatrix} $$
In order to rotate around the x-axis in the viewing space, one needs to think about when to apply the above transformation in the chain of transformations that is applied to the 3D model vertices $\mathbf{X}_{\mathrm{3d}}$ in the employed rendering pipeline.
A typical transformation chain is:
$$ \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}} $$
Let's split the camera tranformation into its rotational and translational part: $$ \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{camera\ transform}}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}} $$
Then we need to apply the x-rotation in-between the camera rotation and translation, because we want to have the rotation of the camera applied beforehand, but not the camera's translation:
$$ \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{new\ camera\ transform}}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}} $$
This would achieve the desired rotation, but typically we do not want to change the camera transformation but the the mesh transformation. This is achieved by $$ \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\ \Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,(\mathtt{T}_{\mathrm{cam\ rot}}\,\mathtt{T}_{\mathrm{cam\ rot}}^{-1})\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\ \Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,\underbrace{\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}}_{\mathtt{T}_{\mathrm{new\ mesh\ transform}}}\,\mathbf{X}_{\mathrm{3d}} $$ Therefore, the short answer to your question is:
Left-multiply the current transformation matrix of your model with $\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}$.
I have generated an "cow" example scene in the GSN Composer that can be used for interactive testing in a WebGL-capable browser:
To get the axis of rotation you could multiply the x-axis by the inverse of the view matrix:
$b = V^{-1} e_1$
Where $b$ is the axis of rotation in world space, $V$ is the view matrix and $e_1$ is the x-axis unit vector.
Now, to make the object to spin around the axis of rotation $b$, we have to translate the object back to the origin, spin it, and then translate it back to its position in world space.
Suppose you have a translation matrix $T$ that send the origin point to the object's location. The translated rotation (spin) of the object $M$ would be:
$M = T R T^{-1}$
Where $R$ is the rotation matrix around the $b$ axis. The matrix $R$ could be obtained using the Rodrigues formula or from a Quaternion or through some other methods out there.
Finally you need to multiply the matrix $M$ to the existing transformation matrix of the object $W$. $W' = M W$ and use $W'$ for transforming the object.
