First of all, note that it follows from the continuity of the sample paths of Brownian motion that $B_{\tau} \in \{-a,b\}$. Since $(B_t)_{t \geq 0}$ is a martingale, the optional stopping theorem shows that $(B_{t \wedge \tau})_{t \geq 0}$ is a martingale, and therefore we have in particular
$$\mathbb{E}(B_{t \wedge \tau}) = \mathbb{E}(B_0) =0.$$
As $|B_{t \wedge \tau}| \leq \max\{a,b\}$ we can apply the dominated convergence theorem to let $t \to \infty$:
$$\mathbb{E}(B_{\tau}) = 0,$$
which means that
$$-a \mathbb{P}(B_{\tau}=-a) + b \mathbb{P}(B_{\tau}=b)=0.$$
On the other hand, we know that
$$\mathbb{P}(B_{\tau}=-a) + \mathbb{P}(B_{\tau}=b)=1.$$
Solving this linear system we get
$$\mathbb{P}(B_{\tau}=-a) = \frac{b}{b+a} \qquad \mathbb{P}(B_{\tau}=b) = \frac{a}{a+b}. \tag{1}$$
Using a similar reasoning as in the first part of the proof and the fact that $(B_t^3-3tB_t)_{t \geq 0}$ is a martingale, we find from the optional stopping theorem that
$$\mathbb{E}(B_{\tau}^3) = 3 \mathbb{E}(B_{\tau} \tau).$$
By $(1)$, this means that
$$\mathbb{E}(B_{\tau} \tau) = \frac{(-a)^3}{3} \frac{b}{a+b} + \frac{b^3}{3} \frac{a}{a+b},$$
i.e.
$$(-a) \mathbb{E}(\tau 1_{\{B_{\tau}=-a\}}) + b \mathbb{E}(\tau 1_{\{B_{\tau}=b\}}) = \frac{ab}{3(a+b)} \left(-a^2+b^2 \right). \tag{2}$$
On the other hand, we have
$$\mathbb{E}(B_{\tau}^2) = \mathbb{E}(\tau)$$
(apply optional stopping to $(B_t^2-t)_{t \geq 0}$) and so
$$a^2 \frac{b}{a+b} + b^2 \frac{a}{a+b} = \mathbb{E}(\tau) = \mathbb{E}(\tau 1_{\{B_{\tau}=-a\}}) + \mathbb{E}(\tau 1_{\{B_{\tau}=b\}}). \tag{3}$$
Equation $(2)$ and $(3)$ are a system of linear equations for
$$x:= \mathbb{E}(\tau 1_{\{B_{\tau}=-a\}}) \quad \text{and} \quad y := \mathbb{E}(\tau 1_{\{B_{\tau}=b\}})$$
which I leave to you to solve; using this information you can calculate $\mathbb{E}(\tau B_{\tau}^2)$. Finally, by another application of the optional stopping theorem,
$$\mathbb{E}(\tau^2) = - \frac{1}{3} \mathbb{E}(B_{\tau}^4) + 2 \mathbb{E}(\tau B_{\tau}^2).$$
Since $(1)$ gives $\mathbb{E}(B_{\tau}^4)$ and we have already calculated $\mathbb{E}(B_{\tau} \tau^2)$, we can finally compute $\mathbb{E}(\tau^2)$.