One of the problems in the book, I am quite confused to show this in a general way. How could I show that $x^2+y^2 = 3(u^2+v^2)$ has no nontrivial integral solutions?
It seems like these problems have a general way to show there are solutions or no solutions to them, and I can not do this yet as I just started to learn about Diophantine equations. Helping me understand the way to solve these problems would be greatly appreciated.
There is no general way.
Rather, there is a variety of known methods and tricks which apply to various types of diophantine equations, and which give you a set of potential strategies. Experience and pattern matching provides the intuition as to which methods are likely to be useful for a given diophantine equation.
For the given equation, as shown below, analyzing it mod $3$ allows a key reduction . . .
Suppose there is a solution $(x,y,u,v)$ such that $x,y,u,v$ are integers, not all zero.
Of all such solutions, choose one such that $x^2 + y^2$ is least.
Since $x,y,u,v$ are not all zero, it follows that $x^2 + y^2$ is a positive integer.
Note that squares are congruent to $0$ or $1$ mod $3$. \begin{align*} \text{Then}\;\;&x^2 + y^2 = 3(u^2+v^2)\\[4pt] \implies\;&3|(x^2 + y^2)\\[4pt] \implies\;&x^2 + y^2 \equiv 0\;(\text{mod}\;3)\\[4pt] \implies\;&x^2 \equiv 0\;(\text{mod}\;3)\;\;\text{and}\;\;y^2 \equiv 0\;(\text{mod}\;3)\\[4pt] \implies\;&3|x\;\,\text{and}\;\,3|y\\[4pt] \implies\;&9|x^2\;\,\text{and}\;\,9|y^2\\[4pt] \implies\;&9|(x^2 + y^2)\\[4pt] \implies\;&9|\bigl(3(u^2 + v^2)\bigr)\\[4pt] \implies\;&3|(u^2 + v^2)\\[4pt] \implies\;&u^2 + v^2 \equiv 0\;(\text{mod}\;3)\\[4pt] \implies\;&u^2 \equiv 0\;(\text{mod}\;3)\;\;\text{and}\;\;v^2 \equiv 0\;(\text{mod}\;3)\\[4pt] \implies\;&3|u\;\,\text{and}\;\,3|v\\[4pt] \end{align*} Hence we can write $$ \begin{cases} x = 3x_1 \qquad\;\;\;\; \\[4pt] y = 3y_1\\[4pt] u = 3u_1\\[4pt] v = 3v_1\\ \end{cases} $$ for some integers $x_1,y_1,u_1,v_1$.
But then \begin{align*} &x^2 + y^2 = 3(u^2 + v^2)\\[4pt] \implies\;&(3x_1)^2 + (3y_1)^2 = 3((3u_1)^2 + (3v_1)^2)\\[4pt] \implies\;&x_1^2 + y_1^2 = 3(u_1^2 + v_1^2)\\[4pt] \end{align*} so we have a solution $(x_1,y_1,u_1,v_1)$ with $$x_1^2 + y_1^2 < x^2 + y^2$$ contrary to the choice of $(x,y,u,v)$.
It follows that the given equation has no nontrivial integer solutions.
HINT:
This is a homogenous equation. If you have a non-trivial solution, you also have a solution with $\gcd(x,y,u,v)=1$. The important fact now is the following: if $a^2+b^2$ is divisible by $3$ then both $a$, $b$ are divisible by $3$. This can be proved by noticing that if $c$ is not divisible by $3$, then $c^2\equiv 1 \bmod{3}$. So assume you have a solution with $\gcd(x,y,u,v)=1$. Since $x^2 + y^2\equiv 0 \bmod{3}$, both $x$, $y$ are divisible by $3$, that is $x = 3 x_1$, $y = 3 y_1$. We get $$3(x_1^2 + y_1^2) = u^2 + v^2$$ and so both $u$, $v$ are divisible by $3$. But then we get $3 \mid \gcd(x,y,u,v)$, contradiction.
The most obvious feature of this equation is that we have a sum of two squares which is a multiple of $3$. Let's think about that. If a sum of two squares $a^2+b^2$ is a multiple of $3$, then $a$ and $b$ each have to be a multiple of $3$ (to see this, just work modulo $3$). Here $x^2+y^2$ is a multiple of $3$, so:
$$x^2+y^2=3(u^2+v^2)$$ $$(3x')^2+(3y')^2=3(u^2+v^2)$$ $$3(x'^2+y'^2) = u^2+v^2$$
At this point you should start to sense that something fishy is going on, especially if you know about the method of infinite descent.
If you want to solve this for yourself, you should stop reading this answer now, read about infinite descent if you haven't already, and take the above as a hint. Otherwise the full solution is below.
Let $A$ denote the set of all pairs $(x, y)$ such that $x^2+y^2$ is $3$ times a sum of two squares. Then the above calculation shows that if $(x, y)\in A$, we also have $(u, v)\in A$, and unless $(u, v)=(x,y)=(0,0)$, we must have $u^2+v^2<x^2+y^2$. So if there is some non-zero pair $(x, y)\in A$, then from it we can spawn an infinite descent of non-zero pairs $(u_i, v_i)$ which is strictly decreasing in euclidean norm. This is not possible, so therefore there is no non-zero pair $(x, y)$.
