In this answer, the user @123 has claimed that by solving the system
$$\begin{cases}a^3 + 39 ab^2 - 18 = 0 \\ 3a^2 b + 13 b^3 - 5 = 0\end{cases}$$
we give $ a = \dfrac 32$ and $ b = \dfrac12$. Could anyone explain for me that how one can solve such a system, please?
You can try eliminating the $a^3$ term, by multiplying the first equation by $b$ and the second equation by $\frac{a}{3}$ and subtracting the two.
$$ \begin{cases} b \left(a^3 + 39 ab^2 - 18 = 0 \right) \\ \frac{a}{3} \left( 3a^2 b + 13 b^3 - 5 = 0 \right) \end{cases} \Rightarrow $$
$$ \begin{cases} b a^3 + 39 ab^3 - 18b = 0 \\ b a^3 + \frac{13}{3} a b^3 - \frac{5}{3} a = 0 \end{cases} \Rightarrow \frac{104}{3} a b^3 + \frac{5}{3} a - 18 b = 0 $$
$$ a= \frac{54 b}{104 b^3 + 5} $$
Substituting $a$ into the second equation yields
$$ \frac{8748 b^3}{(104 b^3+5)^2} + 13 b^3 -5 =0$$ which is solved with $$b^3 = \frac{1}{8} \Rightarrow b = \frac{1}{2} $$
finally you get
$$ \begin{cases} a = \frac{3}{2} \\ b = \frac{1}{2} \end{cases} $$
Setting $a=bt$ then we get $$b^3+t^3+39b^3t-18=0$$ $$3b^3t^2+13b^3-5=0$$ eliminating $b^3$ then we get $$5t^3-54t^2+195t-234=0$$ one solution is $t=3$ then you will get $a=3b$ plugging this in the given equation we get $$a=\frac{3}{2},b=\frac{1}{2}$$
Disclaimer: This answer is probably not appropriate considering (algebra-precalculus) tag, but I'll write it anyway since it might be useful to others stumbling upon this question.
The system is easier to solve when considering original equation $$(a+b\sqrt{13})^3=18+5\sqrt{13}.$$
If we look at norm $N(x+y\sqrt{13}) = x^2-13y^2$, it turns out that $N(18+5\sqrt{13}) = -1$. By multiplicativity of $N$ it follows that $N(a+b\sqrt{13}) = -1$, which gives us $a^2-13b^2 = -1$. Substitute $13b^2 = a^2 + 1$ in $a^3+39ab^2 - 18 = 0$ to get equation $$4a^3+3a-18 = 0.$$
By rational root theorem, the only rational root of the last equation is $a=\frac 32$. It follows that $13b^2 = (3/2)^2+1$, or $b^2 = \frac 14$. From the equation $3a^2b + 13b^3 = 5$, we know that $b$ must be positive, so $b = \frac 12$.
Note that there are complex solutions as well.
Here is one approach. Note from the second equation you have $$ 3a^2 = 5/b - 13b^2 $$ which we can substitute into the first to get $$ 18 = a\left(a^2+39b^2\right) = a\left(\frac{5}{3b} - \frac{13b^2}{3}+39b^2\right) $$ so $$ 3a^2 = \left(\frac{18\cdot 3}{\frac{5}{3b} - \frac{13b^2}{3}+39b^2} \right)^2 $$ which implies an equation in $b$, $$ \frac{5}{b} - 13b^2 = \left(\frac{18\cdot 3}{\frac{5}{3b} - \frac{13b^2}{3}+39b^2} \right)^2, $$ since both sides are equal to $3a^2$.
