I need to prove that every finite subgroup of $SL_2(\mathbb{Q})$ is a subgroup of one of the following groups: $D_3, D_4, D_6$.
Let $G$ is a finite subgroup of $SL_2(\mathbb{Q})$, $ g\in G$. I can define dot product $(x,y)_G=\sum\limits_{g\in G}(gx,gy)$. Why $G$ is a subgroup of $O_2(\mathbb{R})$ and $D_n$? And why $\mathrm{tr}\,g$ is rational? Thanks.
This is not an easy problem with an easy answer. Maybe I've missed the easy way to do it, but here's what I have:
Once you have a dot product, $O_2(\mathbb R)$ are exactly those matrices that preserve the dot product, i.e., those that satisfy $(gx, gy)_G = (x, y)_G$. To see that elements $g \in G$ have this property look at the definition $$(gx, gy)_G = \sum_{g' \in G}(g'gx, g'gy)$$ and note that multiplication by $g$ permutes the elements of the group $G$. So when I look at $g'g$ as $g'$ ranges over all elements of $G$, what I'm getting is just all elements of $G$, which is exactly what I would get if I just looked at $g'$ as $g'$ ranged over all elements of $G$. Thus $$\sum_{g' \in G}(g'gx, g'gy) = \sum_{g' \in G}(g'x, g'y) = (x, y)_G.$$ You can also check that this dot product is positive definite. That means there is an orthonormal basis with respect to this dot product, which means that we can conjugate our finite subgroup into $O_2(\mathbb R)$.
Note that the conjugating element may have entries in $\mathbb R$ so we no longer know that the entries in our matrices are rationals. But conjugation doesn't change the trace of a matrix, and a matrix with rational entries obviously has a rational trace. So while our matrices might have entries from $\mathbb R$ we at least know that the trace is in $\mathbb Q$.
Now presumably you have already seen that elements of $O_2(\mathbb R)$ are simply reflections and rotations. It is known that a finite subgroup of such a group is necessarily either dihedral or cyclic, so in any case is a subgroup of a dihedral group (see here, if you don't understand why this is so you should really ask an entirely new question about it, otherwise this answer is going to be unmanageably long).
Now we know that $G$ has been conjugated into a subgroup of some $D_n$. All we have to do is narrow down which $D_n$ are possible. The rotation matrix for a rotation of angle $\theta$ is $$\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$$ and the trace of such a matrix is $2\cos\theta$. Note that elements in our finite group have finite order so some multiple of $\theta$ is $2\pi$. Thus $\theta$ is a rational multiple of $\pi$. So we need to know for which rational multiples of $\pi$ is $\cos(\theta)$ rational. A slight modification of the accepted answer here shows that this happens only if $\cos(\theta) \in \pm\{0, 1, \frac12\}$, i.e., if and only if $\theta$ is a multiple of $\frac{\pi}{3}$ or $\frac{\pi}{2}$.
These rotations define the dihedral groups $D_n$ for $n = \frac{2\pi}{\theta} = 4, 6$. So your finite subgroup of $\mathrm{SL}_2(\mathbb Q)$ must be a subgroup of $D_4$ or $D_6$. Note that $D_3$ is a subgroup of $D_6$, so it's inclusion in your list was superfluous.
