There may be some confusion.
The class of matrices for which
$$
\mathbf{U}^{*} \mathbf{U} = \mathbf{U} \, \mathbf{U}^{*} = \mathbf{I}
$$
are called unitary.
In general, use of the pseudoinverse matrix implies the classical matrix inverse does not exist. (When the classic inverse exists, it is equal to the pseudoinverse).
To address the concerns of @Michael May:
The Moore-Penrose pseudoinverse is defined and manipulated in the following references.
Your specific question seems to be when does the transpose of a matrix, $\mathbf{A}^{*}$, equal the pseudoinverse of the matrix, $\mathbf{A}^{+}$?
$$
\mathbf{A}^{*} = \mathbf{A}^{+} \, \mathbf{U}^{*} = \mathbf{I}
$$
Given a matrix $\mathbf{A}^{*}\in\mathbb{C}^{m\times n}_{\rho}$, with $\rho \le \min(m,n)$, the singular value decomposition can be expressed in terms of the fundamental subspaces as:
$$
\begin{align}
\mathbf{A} =
\mathbf{U} \, \Sigma \, \mathbf{V}^{*}
%
=
% V
\left[ \begin{array}{cc}
\color{blue}{\mathbf{U}_{\mathcal{R}}} & \color{red}{\mathbf{U}_{\mathcal{N}}}
\end{array} \right]
% Sigma
\left[ \begin{array}{cc}
\mathbf{S} & \mathbf{0} \\ %fixed typo here
\mathbf{0} & \mathbf{0}
\end{array} \right]
% U*
\left[ \begin{array}{c}
\color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} \\
\color{red}{\mathbf{V}_{\mathcal{N}}}^{*}
\end{array} \right] \\
%
\end{align}
$$
The adjoint matrix is then
$$
\begin{align}
\mathbf{A}^{*} =
\mathbf{V} \, \Sigma^{T} \, \mathbf{U}^{*}
%
=
% V
\left[ \begin{array}{cc}
\color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}}
\end{array} \right]
% Sigma
\left[ \begin{array}{cc}
\mathbf{S}^{T} & \mathbf{0} \\
\mathbf{0} & \mathbf{0}
\end{array} \right]
% U*
\left[ \begin{array}{c}
\color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\
\color{red}{\mathbf{U}_{\mathcal{N}}}^{*}
\end{array} \right] \\
%
\end{align}
$$
Because the singular values are real, and the matrix is diagonal,
$$
\mathbf{S} = \mathbf{S}^{T}
$$
The Moore-Penrose pseudoinverse is
$$
\begin{align}
\mathbf{A}^{+} =
\mathbf{V} \, \Sigma^{-1} \, \mathbf{U}^{*}
%
=
% V
\left[ \begin{array}{cc}
\color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}}
\end{array} \right]
% Sigma
\left[ \begin{array}{cc}
\mathbf{S}^{-1} & \mathbf{0} \\
\mathbf{0} & \mathbf{0}
\end{array} \right]
% U*
\left[ \begin{array}{c}
\color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\
\color{red}{\mathbf{U}_{\mathcal{N}}}^{*}
\end{array} \right] \\
%
\end{align}
$$
When does $\mathbf{A}^{*} = \mathbf{A}^{+}$? That is, when does
$$
% V
\left[ \begin{array}{cc}
\color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}}
\end{array} \right]
% Sigma
\left[ \begin{array}{cc}
\mathbf{S} & \mathbf{0} \\
\mathbf{0} & \mathbf{0}
\end{array} \right]
% U*
\left[ \begin{array}{c}
\color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\
\color{red}{\mathbf{U}_{\mathcal{N}}}^{*}
\end{array} \right]
%
=
% V
\left[ \begin{array}{cc}
\color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}}
\end{array} \right]
% Sigma
\left[ \begin{array}{cc}
\mathbf{S}^{-1} & \mathbf{0} \\
\mathbf{0} & \mathbf{0}
\end{array} \right]
% U*
\left[ \begin{array}{c}
\color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\
\color{red}{\mathbf{U}_{\mathcal{N}}}^{*}
\end{array} \right] ?
%
$$
The equality is satisfied when
$$
\mathbf{S} = \mathbf{S}^{-1} = \mathbf{I}_{\rho}
$$
References:
- Pseudoinverse - Interpretation
- What forms does the Moore-Penrose inverse take...
- Singular value decomposition proof
- What is the SVD of A−1?
- Pseudo Inverse Solution for Linear Equation System Using the SVD
