I'll add another answer to the very nice answers above.
First recall the definition of the Lie algebra of a matrix Lie group:
Let $G$ be a matrix Lie group. The Lie algebra of $G$, denoted
$\mathfrak{g}$, is the set of all matrices $X$ such that $e^{tX}$ is in $G$ for all real numbers
$t$.That is, the Lie algebra of a matrix Lie group $G$ is
$
\mathfrak{g}=\{ X\in M_n(\mathbb{C}) : ~ e^{tX} \in G, ~\text{for all}~ t\in \mathbb{R} \} \subseteq M_n(\mathbb{C}).
$
We're interested in finding the Lie algebra of the generalized special orthogonal group $\mathrm{SO}(n;k)$, which is defined as follows: $
\mathrm{SO}(n;k) = \{ A \in M_{n+k}(\mathbb{R}) ~|~A^{tr} g A = g ~~\text{and}~ \det(A)=1 \},
$
where
$
g=\left(
\begin{array}{cccc}
I_n & 0 \\
0 & -I_k
\end{array}
\right).
$
One can prove without too much trouble that if $G$ is a subgroup of $\mathrm{GL}(n, \mathbb{R})$, then the Lie algebra of $G$ must consist
entirely of real matrices. With this in mind, we'll compute the Lie algebra in question in the following proposition.
Proposition.
The Lie algebra of the real symplectic group $\mathrm{SO}(n; k)$ is given by
$
\{ X\in M_{n+k}(\mathbb{R}): ~g X^{tr} g = -X ~~\text{and}~\mathrm{trace}( X)=0\}.$
It is often called the special generalized orthogonal algebra and denoted $\mathfrak{so}(n;k)$.
Proof. Let $X$ be in $\mathfrak{so}(n;k)$ (as noted above, $X$ must be a real matrix). Then, $e^{tX}$ is in $\mathrm{SO}(n;k)$ for all real $t$ so that
$
\begin{array}{llllllllll}
& (e^{tX})^{tr} g e^{tX} &=&g,\\
\text{which implies} & (e^{tX^{tr}}) g e^{tX} &=&g,\\
\text{which implies} & g^{-1} (e^{tX^{tr}}) g e^{tX} &=&I,\\
\text{which implies} & (e^{t g^{-1}X^{tr}g}) e^{tX} &=&I,\\
\text{which implies} & (e^{t g^{-1}X^{tr}g}) &=&e^{-tX}.
\end{array}
$
Differentiating the last equation at $t=0$ yields $g^{-1} X^{tr} g = -X$. This also implies $\mathrm{trace}(X)=\mathrm{trace}(-X)$, so that $\mathrm{trace}(X)=0$. Further, since
$g^{-1} =g$, we have $gX^{tr} g = -X$. Hence, $\mathfrak{so}(n;k)\subseteq \{ X\in M_{n+k}(\mathbb{R}): ~g X^{tr} g = -X ~~\text{and}~\mathrm{trace}( X)=0\ \}$.
Now let $X$ be a real $(n+k) \times (n+k)$ traceless matrix such that $g X^{tr} g = -X$. Since $\mathrm{trace}(X)=0$, then,
$\det(e^{tX})=e^{t\cdot\mathrm{trace}(X)}=1$. And, since $g^{-1}=g$, $g^{-1} X^{tr} g = -X$ and for all real $t$ we have
$
\begin{array}{llllllllll}
g^{-1}(e^{tX} )^{tr} g e^{tX} & =&e^{tg^{-1}X^{tr} g} e^{tX} \\
&=& e^{-tX}e^{tX} \\
&=&I,
%& (e^{tX})^{tr} J e^{tX} &=&J,\\
%\text{if and only if} & (e^{tX^{t}}) J e^{tX} &=&J,\\
%\text{if and only if} & J^{-1} (e^{tX^{t}}) J e^{tX} &=&I,\\
%\text{if and only if} & (e^{t J^{-1}X^{t}J}) e^{tX} &=&I,\\
%\text{if and only if} & (e^{t J X^{t}J}) &=&e^{-tX}.
\end{array}
$
so that $(e^{tX} )^{tr} g e^{tX} =g$, for all real $t$ so that $X\in \mathfrak{so}(n;k)$.
This implies $ \{ X\in M_{n+k}(\mathbb{R}): ~g X^{tr} g = -X~~\text{and}~\mathrm{trace}( X)=0\ \}\subseteq \mathfrak{so}(n;k)$,
and hence $ \{ X\in M_{n+k}(\mathbb{R}): ~g X^{tr} g = -X ~~\text{and}~\mathrm{trace}( X)=0\ \}= \mathfrak{so}(n;k)$.
$\Box$