Many authors use $\mathbf{J_n}$ to represent the $n\times n$ matrix of all ones entries. Since the size $n\times n$ is unchanged throughout this problem, we will omit the subscript:
$$ \mathbf J = \begin{pmatrix} 1 & 1 & \cdots & 1 \\
1 & 1 & \cdots & 1 \\
\vdots & \vdots & \ddots & \vdots \\
1 & 1 & \cdots & 1 \end{pmatrix} $$
Let $\mathbf{x} = \begin{pmatrix} x_1 & x_2 & \cdots & x_n \end{pmatrix}^T$, $\mathbf{y} = \begin{pmatrix} y_1 & y_2 & \cdots & y_n \end{pmatrix}^T$, and $\mathbf{1} = \begin{pmatrix} 1 & 1 & \cdots & 1 \end{pmatrix}^T$, so that $\mathbf J = \mathbf 1 \mathbf 1^T$ and:
$$ \mathbf A = \mathbf J - \mathbf I + \mathbf x \mathbf y^T $$
where $\mathbf I$ is the $n\times n$ identity matrix.
The well-known matrix determinant lemma tells us:
$$ \det \mathbf A = \det(\mathbf J - \mathbf I) (1 + \mathbf y^T (\mathbf J - \mathbf I)^{-1} \mathbf x ) $$
Fortunately the key "ingredients" here have already been worked out in this highly up-voted previous Question, namely:
$$ \det(\mathbf J - \mathbf I) = (-1)^{n-1} (n-1) $$
$$ (\mathbf J - \mathbf I)^{-1} = \left( \frac{1}{n-1} \right) \mathbf J - \mathbf I $$
The interested Reader might care to work out the first of these either from the aforementioned matrix determinant lemma or from evaluation of the characteristic polynomial of $\mathbf J$. The second of these can be verified by multiplying things out, and it can be derived by the Woodbury matrix identity.
Substituting these ingredients in the preceding expression gives:
$$ \begin{align*} \det A &= \det(\mathbf J - \mathbf I ) (1 + \mathbf y^T (\mathbf J - \mathbf I)^{-1} \mathbf x ) \\
&= (-1)^{n-1} (n-1) \left(1 + \mathbf y^T \left[\left(\frac{ 1}{n-1} \right) \mathbf J - \mathbf I \right] \mathbf x \right) \\
&= (-1)^{n-1} \left(n-1 + \mathbf y^T \left[ \mathbf J - (n-1)\mathbf I\right] \mathbf x \right) \\
&= (-1)^{n-1} \left(n-1 + \mathbf y^T \left[\left(\sum x_i \right) \mathbf 1 - (n-1) \mathbf x \right] \right) \\
&= (-1)^{n-1} \left(n-1 + \left[\left(\sum x_i \right) \left(\sum y_i \right) - (n-1) \mathbf y^T \mathbf x \right] \right) \\
&= (-1)^{n-1} \left[\left(\sum x_i \right) \left(\sum y_i \right) + (n-1) (1 -\mathbf y^T \mathbf x) \right]
\end{align*} \\ $$