Reduce the augmented matrix:
$$
\left[\begin{array}{ccc|c}
2&3&1&7\\
3&7&-6&-2\\
5&8&1&\lambda
\end{array}\right]
\sim
\left[\begin{array}{ccc|c}
1&0&5&11\\ 0&1&-3&-5\\ 0&0&0&15 - \lambda
\end{array}\right].
$$
Therefore, the only possible $\lambda$ that works is $\lambda = 15$, and it does in fact work: if $\lambda = 15$, then $(7,-2,15) = (-5z + 11)(2,3,5)+(3z-5)(3,7,8)+z(1,-6,1)$ where $z$ can be any number.
Why does this work? Reducing a matrix doesn't change the set of solutions, so this justifies the reduction. The system is inconsistent if the augmented matrix has a row of the form $\left[\begin{array}{cccc|c}0 & 0 & \cdots & 0 & x \end{array}\right]$ for some $x \neq 0$. So, we must have $\lambda = 15$. For $\lambda = 15$, the matrix has the last row as zero, with one free variable. For a vector $(x,y,z)$ to be a solution of the system means that $x + 5z = 11$ and $y -3z = -5$. This can be rewritten as $x = -5z + 11$ and $y = 3z - 5$. In other words, $z$ is free to be any number and $z$ determines $x$ and $y$ by those two equations. So any vector of the form $v = (-5z+11, 3z-5, z)$ is a solution. By definition of matrix-vector multiplication, $Av$ is a linear combination of the columns of $A$ with the coefficients coming from $v$. Therefore, we can write out the equation $Av = (7,-2,15)$ as $(-5z + 11)(2,3,5)+(3z-5)(3,7,8)+z(1,-6,1) = (7,-2,15)$.