EDIT : Thanks to Andre, he gave me a slap in the face right when I needed it, i.e. before my applied analysis exam tomorrow. XD I'll make this answer right before I bring anymore shame on me.
My ex-approach for the case $Y=5$ is indeed making everything more complicated. The more easy and non-complicated way to do this is just use the definition of the expectation :
$$
\mathbb E(X \, | \, Y = 5) = \sum_{n=1}^{\infty} \, n \, \mathbb P(X = n \, | \, Y = 5).
$$
Now we just need to compute those probabilities. We know that $P(X = 5 \, | \, Y = 5) = 0$, so that removes this term. For the first four terms, note that
$$
\mathbb P(X = n \, | \, Y = 5) = \left( \frac 45 \right)^{n-1} \left( \frac 15 \right) = \frac{4^{n-1}}{5^n}, n=1, 2, 3, 4.
$$
because the condition $Y=5$ only means for $X$ that the first four rolls cannot take the value $5$, hence the value of those rolls become independent and uniformly distributed over $\{1,2,3,4,6\}$.
For the next rolls, $Y=5$ gives information on the first five rolls but no information on the ones after. Thus
$$
\mathbb P(X = n \, | \, Y = 5) = \left( \frac 45 \right)^4 \left( \frac 56 \right)^{n-6} \left( \frac 16 \right).
$$
The $n-6$ stands for the number of rolls after the first five rolls which are not a 6 before you actually get your first $6$ (which gives the $1/6$ term). Therefore, the series we look at in the expectation can be computed, after the first 5 terms, as the derivative of a geometric series in $\left( \frac 56 \right)$. I don't wanna compute it right now because I am going to mess it up, I am definitively too tired for this.