$$\frac{1}{2\pi i} \int_{c-i\infty}^{c+i\infty}\frac{x^{s+1}}{s(s+1)}(1-2^{-s})\zeta^2(s)ds$$
I'm trying to figure out how to approach the problem of deriving an asymptotic expansion of this integral as $x\xrightarrow{}\infty$.
I'm not really looking for an answer so much as a point in the right direction. Thanks.
You don't specify, but I'll assume that initially $c$ is a relatively large positive number somewhere in the domain of absolute convergence, to the right of all poles of the integrand. In fact, I'll assume that $c = 2$.
For simplicity, let's assume the Riemann Hypothesis. This implies the Lindelof Hypothesis, which asserts essentially that $\zeta(\tfrac{1}{2} + it) \ll t^{\epsilon}$ for any $\epsilon > 0$.
Shifting the line of integration to $\mathrm{Re}(s) = \tfrac{1}{2}$ encounters a double pole at $s=1$ from the zeta function, and no other poles.
Then by Cauchy's Residue Theorem, you have that
$$ \begin{align}\frac{1}{2 \pi i}&\int_{2 - i \infty}^{2 + i \infty} \zeta(s)^2 (1 - 2^{-s}) \frac{X^s}{s(s+1)} \\ &= R X \log X + R'X + \frac{1}{2 \pi i}\int_{\frac{1}{2} - i \infty}^{\frac{1}{2} + i \infty} \zeta(s)^2 (1 - 2^{-s}) \frac{X^s}{s(s+1)} \tag{1}\end{align}$$
where $R$ and $R'$ come from the polar portion of the Laurent expansion of $\zeta(s)^2(1 - 2^{-s})/(s(s+1))$.
(Edit to include a bit more detail)
First recall Cauchy's residue theorem, which says that if $f(z)$ is analytic and $\lvert a < 2 \vert$, then $$ f(a) = \frac{1}{2\pi i}\int_{\lvert z \rvert = 2} \frac{f(z)}{z - a} dz.$$ Really, the path of integration can be any simple closed curve containing $a$. There are similar statements for powers $(z-a)^k$ in the denominator relating to the derivative of $f(z)$, but the theme is that the only component of the integrand that actually contributes to the integral are the parts of the Laurent expansion with $(z-a)$ in the denominator.
Since $\zeta(s)^2(1 - 2^{-s})/(s(s+1))$ has a double pole at $s = 1$, we know that its Laurent expansion around $s = 1$ will look like $$ \zeta(s)^2 \frac{(1-2^{-s})}{s(s+1)} = \frac{R}{(s-1)^2} + \frac{R'}{s-1} + g(s)$$ for some analytic function $g(s)$ and constants $R$ and $R'$. Computing these constants isn't hard, but it's slightly aside from this question, so I'll suggest that you look at questions like this one.
On the other hand, $X^s$ has the Taylor expansion
$$ X^s = X + X \log X (s-1) + X (\log X)^2 (s-1)^2/2 + \ldots$$
So there are two ways to get a term of the form $(*)/(s-1)$ in the Laurent expansion of the product $X^s \cdot \zeta(s)^2(1 - 2^{-s})/(s(s+1))$:
From this, one can deduce that the residue integral will contribute $R X \log X + R' X$.
(end edit)
The integral on the right of $(1)$ is in a region of absolute convergence (since $\zeta(1/2 + it) \ll t^{\epsilon}$ By RH while $1/s(s+1)$ decays like $t^{-2}$), and so can be bounded naively by $O(X^{1/2})$.
Thus $$ \frac{1}{2 \pi i}\int_{2 - i \infty}^{2 + i \infty} \zeta(s)^2 (1 - 2^{-s}) \frac{X^s}{s(s+1)} = R X \log X + R' X + O(X^{1/2})$$ for explicitly computable $R$ and $R'$.
On the one hand, this is an answer. But on the other hand, there are also lots of gaps to fill in --- especially if this is your first time thinking about such a Perron-type integral.
I'll also mention that it is possible to improve upon the asymptotic I gave. The dependence on the Riemann Hypothesis was just to simplify arguments --- it's not actually necessary. In fact by an application of the Phragmen-Lindelof Principle and resulting "convexity" bounds for the zeta function, you can show that $\zeta(\epsilon + it) \ll t^{\frac{1}{2} - \epsilon}$ for small $\epsilon > 0$. Further, for small $\epsilon$ this bound is essentially correct, and accurately indicates the rate of growth of $\zeta(s)$.
Inserting into the argument above, you can improve the asymptotic error term to $$ \frac{1}{2 \pi i}\int_{2 - i \infty}^{2 + i \infty} \zeta(s)^2 (1 - 2^{-s}) \frac{X^s}{s(s+1)} = R X \log X + R' X + O(X^{\epsilon})$$ for any $\epsilon > 0$, which is about as good as you could possibly hope for. In particular, note that this is a marked improvement over the analogous results attainable for divisor sums $\sum_{n \leq X} d(n)$ (to which your integral is very closely related).