This is not an answer, but it would be too long to post as a comment.
I don't think really meaningful to talk about the weakest topology in this context; in general topologies on a space are not characterized by their convergent sequences, so such a notion is not well-defined. Even if you get around this problem, the topology you get need not bear any relation with the familiar topologies you can put on $B(X^*)$. In particular, it will likely not even be Hausdorff.
That being said, the following results may help with whatever you're looking for.
A map $T: X^* \rightarrow X^*$ is weak*-weak* continous if and only if there is a norm-norm continuous operator $S : X \rightarrow X$ such that $S^*=T.$ See this answer for a proof. This is equivalent to asserting that $T$ is norm bounded and $T^*(\hat X)\subset \hat X,$ where $\hat{} : X \rightarrow X^{**}$ denotes the canonical embedding. Indeed in that case we can identify $T^*|_{\hat X}$ as a map $X \rightarrow X,$ whose dual map is $T.$
The norm topology on $B(X^*)$ is sufficient. Indeed if $T_n \rightarrow T$ in norm and there is $S_n \in B(X)$ such that $S_n^*=T_n$ for each $n,$ by standard results concerning dual maps $\lVert T_n - T_m \rVert_{B(X^*)} = \lVert S_n-S_m \rVert_{B(X)},$ so $(S_n)$ converges to some $S \in B(X).$ Then a similar argument gives $T_n \rightarrow S^*.$