You already seen that $B = A^{-1} R$, but what is $A$?
Use the rotation $\varphi$ about an axis $\bf v$ formula defined as
$$ A = \mathbf{1}+(\sin\varphi) [\mathbf{v}\times] + (1-\cos\varphi) [\mathbf{v}\times][\mathbf{v}\times]$$
where $[\mathbf{v}\times]$ is the 3×3 skew symmetric cross product operator. It follows that
$$ A^{-1} = \mathbf{1}-(\sin\varphi) [\mathbf{v}\times] + (1-\cos\varphi) [\mathbf{v}\times][\mathbf{v}\times]$$
So now we have an expression for $B$ in terms of the angle $\varphi$
$$ B(\varphi) = R -(\sin\varphi) \mathbf{v}\times R + (1-\cos\varphi) \mathbf{v}\times \mathbf{v}\times R $$
To find $\varphi$ I suggest taking the trace if $B$ and maximizing it. Why? The trace of any 3×3 rotation matrix is: ${\rm trace}(R(\theta))=(1+2\cos\theta)$ with a maximum value at $\theta=0$. By maximizing the trace we are finding rotation $\varphi$ which yields the smallest rotation $B$.
NOTE:
The trace of a combined rotation is
$${\rm trace}( {\rm Rot}(\mathbf{u},\varphi){\rm Rot}(\mathbf{v},\theta)) =\\ n^2+\cos\theta (1-n^2)+\cos\varphi(1-n^2+\cos\theta(1+n^2))-2(\sin\varphi\sin\theta) n$$
where $n=\mathbf{v}\cdot\mathbf{u}$ is the dot product of the two axes.