I'm working on the design of a D6 (6-sided die) based P&P-RPG (pen and paper role-playing game) and am currently looking to balance some mechanics. Now I want to know the formula/chances to roll any X consecutive numbers (like 2, 3, 4) out of a pool of Y D6's.
E.x. A player rolls 5 D6's. What are the chances that he rolls at least 3 consecutive numbers?
I wrote most of this before someone replaced one of your $Y$s by an $X$ in the question, and I used $n$ and $k$ to differentiate between your two uses of $Y$. For the edited question, take $n=Y$ and $k=X$.
You can do this calculation using inclusion–exclusion in two steps.
First let’s find the probability $a_\ell$ to have at least one of each of $\ell$ particular values when you roll $n$ dice. There are $\ell$ conditions, and the number of ways to violate $i$ particular ones of them is $(6-i)^n$, so by inclusion–exclusion the probability to fulfill all of them is
$$ a_\ell=6^{-n}\sum_{i=0}^\ell(-1)^i\binom\ell i(6-i)^n\;. $$
For $k=1$, the probability to have $1$ consecutive number is obviously $1$. (I’m assuming $n\gt0$ throughout).
For $k=6$, the probability to have $6$ consecutive numbers is simply $a_6$, the probability to have $6$ different numbers, which is
\begin{eqnarray} a_6 &=& 6^{-n}\sum_{i=0}^6(-1)^i\binom6i(6-i)^n \\ &=& 6^{-n}\left(6^n-6\cdot5^n+15\cdot4^n-20\cdot3^n+15\cdot2^n-6\right)\;. \end{eqnarray}
For the remaining cases, we can treat $k=3$ to $5$ with a common approach, but we need a separate approach for $k=2$. This is because for $k=2$ it’s possible to have two separate consecutive ranges, e.g. $1,2,3$ and $5,6$, which makes the inclusion–exclusion calculation slightly more involved.
So let’s start with $k=3$ to $5$. There are $7-k$ different ranges of $k$ consecutive numbers, so basically the probability to get at least one of them is $(7-k)a_k$. But that double-counts cases in which there are actually $k+1$ consecutive numbers, so we have to subtract the probability for that, $(7-(k+1))a_{k+1}$. In a typical inclusion–exclusion we’d now have to continue to correct for over-counting of cases with $k+2$ consecutive numbers etc., but you can check that these are already correctly being counted. For instance, each case with $k+2$ consecutive numbers is counted $3$ times in $(7-k)a_k$ and twice in $(7-(k+1))a_{k+1}$, and thus contributes once, as it should. So the probability to get at least $k$ consecutive numbers is
\begin{eqnarray} (7-k)a_k-(7-(k+1))a_{k+1} &=& 6^{-n}\sum_{j=0}^{k+1}(-1)^j\left((7-k)\binom kj-(7-(k+1))\binom{k+1}j\right)(6-j)^n \\ &=& 6^{-n}\sum_{j=0}^{k+1}(-1)^j\left(\binom{k+1}j-(7-k)\binom k{j-1}\right)(6-j)^n\;. \end{eqnarray}
Concretely for the three cases, the probability is
$$ 4a_3-3a_4=6^{-n}\left(6^n-6\cdot4^n+8\cdot3^n-3\cdot2^n\right) $$
for $k=3$,
$$ 3a_4-2a_5=6^{-n}\left(6^n-2\cdot5^n-2\cdot4^n+8\cdot3^n-7\cdot2^n+2\right) $$
for $k=4$ and
$$ 2a_5-a_6=6^{-n}\left(6^n-4\cdot5^n+5\cdot4^n-5\cdot2^n+4\right) $$
for $k=5$.
Now for $k=2$. Here we have $5$ possible ranges of $2$ consecutive numbers. If we select $j$ of them, with $1\le j\le5$, these can be in $m$ overlapping runs, with $1\le m\le j$. (The cases with large values of both $j$ and $m$ don’t actually occur, but that’s taken care of by the corresponding binomial coefficients being zero.) They cover $j+m$ different numbers, which can be chosen in $\binom{j-1}{m-1}\binom{6-j}m$ ways (since we can distribute the $j$ ranges over the $m$ non-empty runs in $\binom{j-1}{m-1}$ ways (see stars and bars) and then we choose $m$ positions for the runs among the $m$ runs and the $6-(j+m)$ remaining numbers). Then by inclusion–exclusion the probability to get $2$ consecutive numbers is
\begin{eqnarray} &&\sum_{j=1}^5(-1)^{j+1}\sum_{m=1}^j\binom{j-1}{m-1}\binom{6-j}ma_{j+m} \\ &=& 6^{-n}\sum_{j=1}^5(-1)^{j+1}\sum_{m=1}^j\binom{j-1}{m-1}\binom{6-j}m\sum_{i=0}^{j+m}(-1)^i\binom{j+m}i(6-i)^n \\ &=& 6^{-n}\left(6^n-4\cdot3^n+2\cdot2^n+2\right) \;. \end{eqnarray}
Here’s the Sage code for that computation:
i,j,l,m,n=var('i,j,l,m,n')
sum((-1)**(j+1) * binomial(j-1,m-1) * binomial (6-j,m) * (-1)**i * binomial(j+m,i) * (6 - i)**n for j in range(1,6) for m in range (1,j+1) for i in range(0,j+m+1))
With hindsight, we could have gotten this with a bit less effort: There are two consecutive numbers unless all numbers are isolated, which is possible in $4$ ways for $3$ numbers, in $10$ ways for $2$ numbers and in $6$ ways for $1$ number, and there are $3^n-3\cdot2^n+3$ results with exactly $3$ particular numbers, $2^n-2$ results with exactly $2$ particular numbers and $1$ result with exactly one particular number, for a total of
$$ 4\left(3^n-3\cdot2^n+3\right)+10\left(2^n-2\right)+6\cdot1=4\cdot3^n-2\cdot2^n-2 $$
results, and the complement yields the probability above.
For your example with $n=5$ and $k=3$, the probability is
$$ 6^{-5}\left(6^5-6\cdot4^5+8\cdot3^5-3\cdot2^5\right)=\frac{145}{324}\approx45\%\;. $$
