Write $X = 3 \sin B + 5 \cos B = \sqrt{34} \sin (B + \alpha)$ (with $\tan \ \alpha = \frac{5}{3}$ but not needed) then $Y = 3\cos B - 5\sin B = \sqrt{34} \cos( B + \alpha)$, then use $\sin^2 \theta + \cos^2 \theta = 1$.
The details of the calculation are: You can write $3 \sin B + 5 \cos B$ as $\sqrt{34} (\frac{3}{\sqrt{34}} \sin B + \frac{5}{\sqrt{34}} \cos B)$.
This is OK and it hasn't changed the result, but why divide and multiply by $\sqrt{34}$? That is a bit of experience, but notice that $3^2 + 5^2 = 9 + 25 = 34$.
But now you can put $\cos \alpha = \frac{3}{\sqrt{34}}$ and $\sin \alpha = \frac{5}{\sqrt{34}}$ (you have $\sin^2 \alpha + \cos^2 \alpha = 1$ so that this is admissible) and the final result for $X$ is $\sqrt{34} \sin ( B + \alpha)$.
After that, you can then look at $Y$ and divide and multiply by 6 again and this time you have $Y = 3 \cos B - 5 \sin B = \sqrt{34} (\frac{3}{\sqrt{34}} \cos B - \frac{5}{\sqrt{34}} \sin B) = \sqrt{34} (\cos \alpha \cos B - \sin \alpha \sin B) = \sqrt{34} \cos( B + \alpha)$.
Because $X^2 + Y^2 = 34$ (from $\sin^2 + \cos^2 = 1)$ then $Y^2 = 9, Y = \pm 3$ since you are given $X = 5$.