It does not matter which solution you choose; the algebra works out in the end. If you choose to let $0 \leq \phi \leq \pi$, then R would actually be negative, not positive, which you can see from equating coefficients. It still gives you the same function however.
Let $\sqrt 3 \sin x - \cos x=R\sin(x+ \phi)$
Expanding the RHS, $\sqrt 3 \sin x - \cos x = R \sin x \cos \phi + R\cos x \sin \phi $
Equating coefficients, then $R\cos \phi= \sqrt 3 (1),\\ R \sin \phi= -1 (2)$
So $\tan \phi =\frac{-1}{\sqrt 3}$
Arbitrarily, we choose the solutions $\phi=\frac{-\pi}{6},\frac{5\pi}{6}$.
If $\phi=\frac{-\pi}{6}$, $\sin \phi =\frac{-1}{2}$
From substitution into $(2)$, $\frac{R}{-2}=-1 \implies R=2$
However, if $\phi=\frac{5\pi}{6}$, $\sin \phi=\frac{1}{2}$.
Substitution into (2) gives $\frac{R}{2}=-1 =-2\implies R=-2$
If you substitute these values in and expand the entire RHS out you will get it to match the LHS regardless of which value you initially chose.