Let $V$ be the subspace of smooth functions spanned by
\begin{align*}
e_1(t) &= \cos(at) & e_2(t)=\sin(bt)
\end{align*}
Let $W$ be the subspace of smooth functions spanned by
\begin{align*}
f_1(t) &= \cos(bt) & f_2(t)=\sin(at)
\end{align*}
Let $d:V\to W$ be the linear operator defined by $df(t)=f^\prime(t)$. Note that
\begin{array}{rcrcrcrc}
de_1(t) & = & -a\sin(at) &=& \color{red}{0}\, f_1(t) &+& (\color{red}{-a})\,f_2(t) \\
de_2(t) &=& b\cos(bt) &=& \color{blue}{b}\,f_1(t) &+& \color{blue}{0}\,f_2(t)
\end{array}
This implies that the matrix representing $d$ in the given bases is
$$
D=
\begin{bmatrix}
\color{red}{0} & \color{blue}{b}\\
\color{red}{-a} & \color{blue}{0}
\end{bmatrix}
$$
as expected.
Here's the general situation:
Let $T:V\to W$ be a linear map and let
\begin{align*}
\alpha &= \{v_1,\dotsc,v_n\} &
\beta &= \{w_1,\dotsc,w_m\}
\end{align*}
be bases for $V$ and $W$ respectively. This means there are unique scalars $\lambda_{ij}$ for $1\leq i\leq m$ and $1\leq j\leq n$ such that
$$
\begin{array}{ccccccccc}
T(v_1) & = & \color{red}{\lambda_{11}}w_1 & + & \color{red}{\lambda_{21}}w_2 & + &\dotsb &+ & \color{red}{\lambda_{m1}}w_m \\
T(v_2) & = & \color{blue}{\lambda_{12}}w_1 & + & \color{blue}{\lambda_{22}}w_2 & + &\dotsb &+ & \color{blue}{\lambda_{m2}}w_m \\
\vdots & \vdots & \vdots &\vdots &\vdots &\vdots &\ddots&\vdots&\vdots \\
T(v_n) & = & \color{green}{\lambda_{1n}}w_1 & + & \color{green}{\lambda_{2n}}w_2 & + &\dotsb &+ & \color{green}{\lambda_{mn}}w_m \\
\end{array}
$$
The matrix of $T$ relative to the bases $\alpha$ and $\beta$ is defined as
$$
[T]_\alpha^\beta=
\begin{bmatrix}
\color{red}{\lambda_{11}} & \color{blue}{\lambda_{12}} & \dotsc & \color{green}{\lambda_{1n}} \\
\color{red}{\lambda_{21}} & \color{blue}{\lambda_{22}} & \dotsc & \color{green}{\lambda_{2n}} \\
\vdots & \vdots & \ddots & \vdots \\
\color{red}{\lambda_{m1}} & \color{blue}{\lambda_{m2}} & \dotsc & \color{green}{\lambda_{mn}}
\end{bmatrix}
$$
Note that this is an $m\times n$ matrix.