"The automorphism of $\mathfrak{g}$ induced by the automorphism of $R$" is actually a bit ambiguous, but it's clear from context that they mean the automorphism that switches $x_1 \leftrightarrow x_3$, $y_1 \leftrightarrow y_3$ and fixes $x_2$ and $y_2$. It's probably a good exercise to check that this indeed defines an automorphism and determines it completely.
In fact, let's switch to Chevalley basis notation and call
- $e_{\alpha_i} := x_i$, $e_{-\alpha_i} := y_i$ for $i=1,2,3$
- $e_{\alpha_1+\alpha_2} := [x_1, x_2]$, $e_{\alpha_2+\alpha_3} := [x_2, x_3]$, $e_{-\alpha_1-\alpha_2} := [y_1, y_2]$, $e_{-\alpha_2-\alpha_3} := [y_2, y_3]$
- $e_{\alpha_1+\alpha_2+\alpha_3} := [x_1, [x_2, x_3]] = [[x_1, x_2], x_3]$, $e_{-\alpha_1-\alpha_2-\alpha_3} := [y_1, [y_2, y_3]] = [[y_1, y_2], y_3]$
where according to what they call an $\mathfrak{sl}_2$-triple in your source you might have to throw in some minus signs in the second line.
In any case, it's a good exercise to recall that these twelve $e$'s (each of them being a basis of the root space to the root in its index), together with $h_1, h_2$ and $h_3$, are a vector space basis of $\mathfrak{g}$, and the one we want to work with: Because it makes the automorphism completey explicit. E.g, it also switches $h_1 \leftrightarrow h_3$, $e_{\alpha_1+\alpha_2} = [x_1,x_2] \leftrightarrow -[x_2,x_3] = - e_{\alpha_2+\alpha_3}$, and fixes $h_2$ as well as $e_{\alpha_1+\alpha_2+\alpha_3}$.
It's linear algebra to see that a vector space basis of the fixed space $\mathfrak{g}^\sigma$ is:
- $h_1+h_3$, $h_2$ (basis of the new, two-dimensional CSA $\mathfrak{h}^\sigma$)
- $e_{\alpha_1}+e_{\alpha_3}$, $e_{\alpha_2}, e_{\alpha_1+\alpha_2}-e_{\alpha_2+\alpha_3}, e_{\alpha_1+\alpha_2+\alpha_3}$ (four positive root vectors)
- $e_{-\alpha_1}+e_{-\alpha_3}, e_{-\alpha_2}$, $e_{-\alpha_1-\alpha_2}-e_{-\alpha_2-\alpha_3}, e_{-\alpha_1-\alpha_2-\alpha_3}$ (four negative root vectors)
Setting $e_{\beta_1} := e_{\alpha_1}+e_{\alpha_3}$ and $e_{\beta_2} := e_{\alpha_2}$, compute how nicely
$[e_{\beta_1}, e_{\beta_2}] = e_{\alpha_1+\alpha_2}-e_{\alpha_2+\alpha_3} =: e_{\beta_1+\beta_2}$
and
$[e_{\beta_1}, e_{\beta_1 +\beta_2}] = -2e_{\alpha_1+\alpha_2+\alpha_3} =: e_{2\beta_1 + \beta_2}$ etc.
so that $\beta_1$ and $\beta_2$ are a a basis of a root system of type $B_2$, with $\beta_1$ being the short root. Check further that $\mathfrak{h}^\sigma$ indeed acts on those new root spaces, with $h_1+h_3$ of course being the coroot of $\beta_1$, and $h_2$ the coroot of $\beta_2$. So $\mathfrak{g}^\sigma$ decomposes as
$\mathfrak{h}^\sigma \oplus $ root spaces, and hence is semisimple of type $B_2$.
It is another exercise to see how this, written out in matrices, modulo some choices (scalings and signs I would guess), is the same as what happens in Andreas Cap's answer, just from a very different viewpoint.