Efficient algorithm to find all unique combinations of set given duplicates

Question

The number of combinations for a 4 choose 2 is 6. But what if I have a set in which 2 of the elements are the same? Of course I could do a unique check for each item but this is too computationally expensive with larger sets (as is calculating all the possible combinations in the first place). Knowing that my sets have many duplicates, I am trying to find an algorithm that can take advantage of this to reduce the amount of work it takes to generate all the combinations I care about.

Normally, [1',1'',2,3] choose 2 yields [1',1''] [1',2] [1',3] [1'',2] [1'',3] [2,3]. I know though that [1',2] == [1'',2] and [1',3] == [1'',3] so the number of combinations I am interested in is only 4.

To be clear, I am trying to generate the actual unique combinations, not just the number of them that exist, and without having to check each generated set against previous sets.

Update Here is a working implementation in javascript using the lodash utility library for convenience:

let _ = require('lodash');
let _items_1 = ['A','A','A','B','B','C'];
let items_1 = {'A':3, 'B':2, 'C':1};
function UniqueCombinations(set, n) {
    let combinations = [];
    let props = Object.getOwnPropertyNames(set);
    for (let p = 0; p < props.length; p++) {
        for (let i = Math.min(set[props[p]], n); i > 0 ; i--) {
            // if (_.sum(_.values(_.pick(set, props.slice(p+1)))) < n-i) continue;
            if (n-i > 0) {
                let rest = UniqueCombinations(_.pick(set, props.slice(p+1)), n-i)
                for (let c = 0; c < rest.length; c++) {
                    let combination = {};
                    if (i > 0) combination[props[p]] = i;
                    Object.assign(combination, rest[c]);
                    combinations.push(combination);
                }
            } else {
                let combination = {};
                combination[props[p]] = i;
                combinations.push(combination);
            }
        }
    }
    return combinations;
}
let combinations = UniqueCombinations(items_1, 3);
console.log(combinations);
combinations.forEach(combination => console.log(_.transform(combination, function(result, value, key) {
    result.push(...key.repeat(value).split(''));
}, [])));

[ { A: 3 }, { A: 2, B: 1 }, { A: 2, C: 1 }, { A: 1, B: 2 }, { A: 1, B: 1, C: 1 }, { B: 2, C: 1 } ]

[ 'A', 'A', 'A' ] [ 'A', 'A', 'B' ] [ 'A', 'A', 'C' ] [ 'A', 'B', 'B' ] [ 'A', 'B', 'C' ] [ 'B', 'B', 'C' ]

This is by no means the cleanest or most performant implementation but a fine starting place for anyone else looking to do this.

Answer 1

Let's imagine our base multiset has $n$ distinct elements (which may as well be $1$ through $n$ for convenience), and multiplicities $m_i$ for $i=1,..n$ describing the repeated appearances of those elements.

The Wikipedia article on Combination says:

To refer to combinations in which repetition is allowed, the terms $k$-selection, $k$-multiset, or $k$-combination with repetition are often used.

Here $k$ represents the total size of the selection, multiset, or "combination with repetition" being formed.

An algorithm to generate the $k$-selections can proceed serially through nested (dependent) choices about how many of the first element, the second element,..., the $n$th element to use, subject at each stage that the $i$th element not repeat more than the available multiplicity $m_i$ of that element and not more than $k$ minus the sum of previously chosen repetitions. There will not be any solutions if $k \gt \sum m_i$, of course, since we want to arrive at exactly $k$ elements.

More formally the chosen repetitions $r_1,r_2,\ldots,r_n$ satisfy:

$$ r_1 + r_2 + \ldots + r_n = k $$

where $0 \le r_i \le m_i$ for all $i=1,..,n$. Such expressions of $k$ are called weak compositions. In this context we should further qualify them as restricted weak compositions, in view of $r_i \le m_i$.

Several representations of the outcomes are possible, and one may be more attractive than another depending on the intended application. For example, we might output a string of length $k$, in which a symbol for "first element" gets repeated $r_1$ times and so forth. We would order all the first element symbols before all the second element symbols, etc. and in this way arrive at a unique string representation. If we choose the counts of individual elements as outlined below, then we will generate these strings in lexicographic order (most first symbols first, etc.).

Another sort of representation dispenses with the need to specify arbitrary symbols for such strings and relies instead on a $2\times n$ array in which all the entries are nonnegative integers. This representation is a contingency table in which the first row will contain values $r_1,r_2,\ldots,r_n$ and the second row will contain $m_1-r_1,m_2-r_2,\ldots,m_n-r_n$.

Notice that the row sums are $k$ and $\sum m_i -k$ respectively, and the column sums are the multiplicities $m_i$ from left to right.

We can work out all the possible (distinct) ways to fill in such a contingency table, and these correspond to all the possible $k$-sub-multisets. Assume that $k \le \sum m_i$ so that at least one $k$-selection is possible.

Start with the leftmost column and choose the maximum possible repetitions $r_1 = \min(m_1,k)$. Then proceed recursively to choose entries in the remaining columns which add up to $k'=k-r_1$. When those possibilities are exhausted, possibly because the remaining $m_i$ are insufficient to total $k'$, backtrack to our choice in the leftmost column and reduce the selected repetitions $r_1$ by one. When $r_1$ reaches $-1$ in the course of this backtracking, quit.

Example Take the case outlined in the Question, of multiplicities $m_1 = 2$ and $m_2 = m_3 = 1$. We ask for $2$-selections from this multiset, $\{\{1,1,2,3\}\}$, i.e. $k=2$.

In the first pass we choose a multiplicity for element $1$ to be $r_1 = \min(m_1,k) = 2$. Since $k'=k-r_1=0$ for the remaining columns, this forces those to be zero. So our first output is this completed contingency table:

$$\begin{array}{|c|c|c|} \hline 2 & 0 & 0 \\ \hline 0 & 1 & 1 \\ \hline \end{array} $$

Subsequently we backtrack to the choice of $r_1$ in the first column and reduce it by one. This yields two possibilities for the remaining columns:

$$\begin{array}{|c|c|c|} \hline 1 & 1 & 0 \\ \hline 1 & 0 & 1 \\ \hline \end{array} $$

$$\begin{array}{|c|c|c|} \hline 1 & 0 & 1 \\ \hline 1 & 1 & 0 \\ \hline \end{array} $$

Finally we again backtrack to the first column and choose $r_1 = 0$, which allows only one solution:

$$\begin{array}{|c|c|c|} \hline 0 & 1 & 1 \\ \hline 2 & 0 & 0 \\ \hline \end{array} $$

Note that we've produced these $2$-selections in lexicographic order if their string representations are considered: 11, 12, 13, and 23.

Answer 2

Generate the actual unique combinations, not just the number of them that exist, and without having to check each generated set against previous sets.

First, we need to remove the repetitions that occur when comparing each set, then we need to formulate how to generate the unique combinations, and lastly we need a way to verify the generated set without actually referring to previous sets.

To accomplish the first, you can use the inclusion-exclusion principle. This will allow you to manipulate any unique combinations through a Cartesian product. To accomplish the second, we need a set of functions in addition to a linear operator that can verify solutions with respect to sequences. This way, we can prove a characteristic equation for your combinations which also create a generator function. You won't need to compare actual sequences and values, because you'll have the generator function and a frame of reference to formally verify the output.

If two sets share elements, this is restricted to the intersection between them. For ${A}$, ${B}$ given elements $(x_1,...,x_n), (y_1,...,y_n)$ the ranges provide the conditions for both unique and equal membership of the sets ${A}, {B}$. Keeping in mind any point in an intersection is therefore not unique, by using the inclusion-exclusion principle:

$ N(A \cup B) = N(A)+N(B) - N(A \cap B)$

you've then isolated the elements unique between each set. This is treated as the range for permutations possible. This subset of sequences has the Cartesian product:

$(f:A \to B)$, $\lbrace{(x,f(x)): x \in A} \subset A \times B\rbrace$

Using concepts of graph theory and a relation $R$ from finite ${A}$ and finite ${B}$ treated as a bipartite graph, we will rely on antisymmetry and comparison for our relation $R$.

Now we introduce a theorem for a recursive function, where if $f$ is recursively defined, $f(n)$ is unique for every $n \in \mathbb{Z^+}$.

Since your goal is for large datasets, we can use the sequences calculated to treat our relation $R$ using the recursive function and recurrences with a generating function. Since any unique $n$ results in a unique $f(n)$, we can solve for a characteristic function, this uses the following theorem concerning a linear homogeneous recurrence relation:

$p(x)$ and $g(x)$ are the characteristic and generating functions respectively, for the linear homogeneous recurrence relations with constant coefficients:

$a_n = c_{1}a_{n-1} + c_{2}a_{n-2} + ... + c_{r}a_{n-r}$ for consecutive initial conditions $a_i = A_i$ $(i = 0,...,r-1)$, then $p(x) = x^r - c_{1}x^{r-1} - c_{2}x^{r-2} -...-c_r$, and $g(x)= \frac {u(x)}{v(x)}$. The denominator is

$v(x) = 1 - c_{1}x - c_{2}x^{2} -...-c_{r}x^r$ and the numerator is

$u(x)= [A_0 + A_{1}x + ... + A_{r-1}x^{r-1}] - (c_{1}x)[A_0 + A_{1}x + ... + A_{r-2}x^{r-2}] - (c_{2}x^2)[[A_0 + A_{1}x + ... + A_{r-3}x^{r-3}] -...- (c_{r-1}x^{r-1})[A_0]$.

We will now define a solution operator, $\Omega_{r_0}^{r}$ which is invertible and a linear operator. Finally we have the basis $f_{1}(r),...,f_{n}(r)$ as a fundamental system of solutions. Given this, you can work with the range and unique sequences to calculate the generator function verified with the solution operator and fundamental system of solutions with respect to the linear homogeneous recurrence relations.

Answer 3

So, complementing "hardmath"'s answer, you are given the multiset $$ \left\{ {\left\{ {1_1 ,1_2 , \cdots ,1_{m_{\,1} } } \right\},\left\{ {2_1 ,2_2 , \cdots ,2_{m_{\,2} } } \right\}, \cdots ,\left\{ {n_1 ,n_2 , \cdots ,n_{\,m_{\,n} } } \right\}} \right\} $$ and you have to generate all the k-multisets $$ \left\{ {\left\{ {1_1 ,1_2 , \cdots ,1_{r_{\,1} } } \right\},\left\{ {2_1 ,2_2 , \cdots ,2_{r_{\,2} } } \right\}, \cdots ,\left\{ {n_1 ,n_2 , \cdots ,n_{\,r_{\,n} } } \right\}} \right\} $$ such that $$ \left\{ \begin{gathered} 0 \leqslant r_{\,k} \leqslant m_{\,k} \hfill \\ r_{\,1} + r_{\,2} + \cdots + r_{\,n} = k \hfill \\ \end{gathered} \right. $$

If the multiplicities were the same , the number of such k-multisets would be computable by the Balls-in-Bins formula. Also we can profitably use that formula when the multiplicities assume a set of a few values.

Concerning their generation, a possible algorithm might be to split $k$ by filling to the max the first ($q$) places and distribute the rest ($c$) among the remaining, starting with $\left\{ {c,0, \cdots 0} \right\}$ $$ \begin{gathered} \begin{array}{*{20}c} {m_{\,1} } & \cdots & {m_{\,q} } \\ \end{array} \;\overbrace {\begin{array}{*{20}c} c & 0 & \cdots & 0 \\ \end{array} }^{tot = c} \hfill \\ \begin{array}{*{20}c} {m_{\,1} } & \cdots & {m_{\,q} } \\ \end{array} \;\overbrace {\begin{array}{*{20}c} {c - 1} & 1 & \cdots & 0 \\ \end{array} }^{tot = c} \hfill \\ \quad \quad \vdots \hfill \\ \begin{array}{*{20}c} {m_{\,1} } & \cdots & {m_{\,q} } \\ \end{array} \;0\;\overbrace {\begin{array}{*{20}c} c & \cdots & 0 \\ \end{array} }^{tot = c} \hfill \\ \quad \quad \vdots \hfill \\ \begin{array}{*{20}c} {m_{\,1} } & \cdots & {m_{\,q} } \\ \end{array} \;\begin{array}{*{20}c} 0 & \cdots & 0 \\ \end{array} \;\overbrace {\begin{array}{*{20}c} \cdots \\ \end{array} }^{tot = c} \hfill \\ \end{gathered} $$ Then we diminuish $c$ by $1$, and distribute the $1$ among the remaining, starting from the first. This in turn gets diminuished by one, etc., until pushing the $1$ to the very end.
Continue by diminuishing $c$ by $2$,$3$ ..$d$, and feed that amount into the remaining places with the same algorithm untill the last place is full ($d=m_n$). Freeze it out and continue to distribute $d+1-m_n$ into the remaining but last.
If $c=0$ freeze it and pass its role to the next "active" place.
End when there is no next un-freezed place.
Restart by putting the first at $m_1-1$ and repeat the same algorithm, with $k-m_1+1$, for the remaining $n-1$. $$ \begin{array}{*{20}c} {m_{\,1} - 1} & \cdots & {m_{\,q'} } \\ \end{array} \;\overbrace {\begin{array}{*{20}c} {c'} & 0 & \cdots & 0 \\ \end{array} }^{tot = c'} $$ End the outer cycle when the active cell has all the preceding cells at 0 and the following ones full.
So it is the same algorithm, which is telescoped in and out.