There are how many ways to select $r$ things from $n$ categories with maximum $k$ repetitions are allowed from each category?
I think its only solvable if and only if $nk\ge r$ and I also believe that it's hard to find a formula for this problem but a generating function may be possible.
Let $c(r,n,k)$ be the number of ways to do this. Consider conditioning on the number of items from the last category. If you select $i$ of these, you still have to select $r-i$ from $n-1$ categories, still with at most $k$ repetitions allowed. Thus we get the recursion $$ \eqalign{c(r,n,k) &= \sum_{i=0}^{\min(r,k)} c(r-i,n-1,k)\cr c(r,0,k) &= 0 \ \text{if}\ r > 0\cr c(0,n,k) &= 1}$$
If $g(x, y, k) = \sum_{r,n \ge 0} c(r,n,k) x^r y^n $ is the generating function (for fixed $k$), this translates to $$ g(x,y,k) = 1 + \sum_{i=0}^k x^i y g(x,y,k) = 1 + \dfrac{1-x^{k+1}}{1-x} y g(x,y,k)$$ and thus $$ g(x,y,k) = \dfrac{1-x}{1 - x - y + x^{k+1} y} $$
