Here are the matrices satisfying the conditions in Dietrich Burde's answer.
The count is $(11\times 4) + (5\times 2) + (2\times 1) = 56$
Depending whether there can be $2$ or $1$ or $0$ exchange along the two diagonals (i.e. when numbers are not equal).
- Matrices having $0,1$ in main diagonal
$\begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 1 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 1 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 1 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 1 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 2 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 2 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 2 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 2 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 3 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 3 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 3 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 3 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 4 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 4 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 4 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 4 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 5 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 5 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 5 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 5 & 0 \end{pmatrix}\quad$
$\begin{pmatrix} 0 & 6 \\ 0 & 1 \end{pmatrix}\quad\begin{pmatrix} 0 & 0 \\ 6 & 1 \end{pmatrix}\quad\begin{pmatrix} 1 & 6 \\ 0 & 0 \end{pmatrix}\quad\begin{pmatrix} 1 & 0 \\ 6 & 0 \end{pmatrix}\quad$
- Matrices having $2,6$ in main diagonal
$\begin{pmatrix} 2 & 5 \\ 1 & 6 \end{pmatrix}\quad\begin{pmatrix} 2 & 1 \\ 5 & 6 \end{pmatrix}\quad\begin{pmatrix} 6 & 5 \\ 1 & 2 \end{pmatrix}\quad\begin{pmatrix} 6 & 1 \\ 5 & 2 \end{pmatrix}\quad$
$\begin{pmatrix} 2 & 6 \\ 2 & 6 \end{pmatrix}\quad\begin{pmatrix} 2 & 2 \\ 6 & 6 \end{pmatrix}\quad\begin{pmatrix} 6 & 6 \\ 2 & 2 \end{pmatrix}\quad\begin{pmatrix} 6 & 2 \\ 6 & 2 \end{pmatrix}\quad$
$\begin{pmatrix} 2 & 4 \\ 3 & 6 \end{pmatrix}\quad\begin{pmatrix} 2 & 3 \\ 4 & 6 \end{pmatrix}\quad\begin{pmatrix} 6 & 4 \\ 3 & 2 \end{pmatrix}\quad\begin{pmatrix} 6 & 3 \\ 4 & 2 \end{pmatrix}\quad$
- Matrices having $3,5$ in main diagonal
$\begin{pmatrix} 3 & 1 \\ 1 & 5 \end{pmatrix}\quad\begin{pmatrix} 5 & 1 \\ 1 & 3 \end{pmatrix}\quad$
$\begin{pmatrix} 3 & 4 \\ 2 & 5 \end{pmatrix}\quad\begin{pmatrix} 3 & 2 \\ 4 & 5 \end{pmatrix}\quad\begin{pmatrix} 5 & 4 \\ 2 & 3 \end{pmatrix}\quad\begin{pmatrix} 5 & 2 \\ 4 & 3 \end{pmatrix}\quad$
$\begin{pmatrix} 3 & 5 \\ 3 & 5 \end{pmatrix}\quad\begin{pmatrix} 3 & 3 \\ 5 & 5 \end{pmatrix}\quad\begin{pmatrix} 5 & 5 \\ 3 & 3 \end{pmatrix}\quad\begin{pmatrix} 5 & 3 \\ 5 & 3 \end{pmatrix}\quad$
$\begin{pmatrix} 3 & 6 \\ 6 & 5 \end{pmatrix}\quad\begin{pmatrix} 5 & 6 \\ 6 & 3 \end{pmatrix}\quad$
* Matrices having $4,4$ in main diagonal
$\begin{pmatrix} 4 & 2 \\ 1 & 4 \end{pmatrix}\quad\begin{pmatrix} 4 & 1 \\ 2 & 4 \end{pmatrix}\quad$
$\begin{pmatrix} 4 & 3 \\ 3 & 4 \end{pmatrix}\quad$
$\begin{pmatrix} 4 & 4 \\ 4 & 4 \end{pmatrix}\quad$
$\begin{pmatrix} 4 & 6 \\ 5 & 4 \end{pmatrix}\quad\begin{pmatrix} 4 & 5 \\ 6 & 4 \end{pmatrix}\quad$