Faster finding of a subset of bits with all combinations in the bitstrings

Question

Assume that I have a bunch of bitsets (strings on $\{0,1\}$) of the same length, e.g.:

101110001
001001101
010101010
101001001
101010101

I want to find the largest set of bits (indices, not necessarily consecutive) that takes all possible values in these bitsets. In the example above:

101|11|0001
001|00|1101
010|10|1010
101|00|1001
101|01|0101

the fourth and the fifth bits take all possible combinations 00 (2nd bitset), 01 (5th), 10 (3rd), 11 (1st), so they fit. Clearly, there is no set of bits of length $3$ that takes all possible values (in this case, we only have 5 bitsets, so it's impossible to get $2^3=8$ combinations), so bits $4$ and $5$ answer the question.

I have a lot of bitstrings ($3 \cdot 10^7$) and bits ($100$), and the answer has $\approx 10$ bits, so brute force is out of the question. Based on how the bitstrings are generated, there is a lot of permutational symmetry, so some simple prunings (e.g. if bit $i$ only takes value $1$ (and never takes value $0$), then we never consider any set of bits containing $i$) probably won't work. What can I do?

As an easier version, I'm also interested in the case when there are $3 \cdot 10^5$ bitstrings, $60$ bits, and the answer is $\approx 8$. For this, I'm currently running the bruteforce, and, assuming that the answer is $8$, I expect that computation to finish in a few days (I know that there is a set of size $8$, and I expect and need to show that there is no set of size $9$).

Answer 1

This problem sounds very hard, especially at the scale you're dealing with.

One approach might be to try using a SAT solver. I don't know whether this will be even remotely feasible to solve with SAT, but you could give it a try.

Let boolean variable $x_{j,k}$ represent whether bit index $j$ is the $k$th one selected (for $j=1,\dots,60$ and $k=1,\dots,8$). Let $y_{s,\ell}$ (for $s \in \{0,1\}^8$ and $\ell=0,\dots,\lg(3 \cdot 10^5)$) be boolean variables defined so that $y_{s,0},\dots,y_{s,\lg(3 \cdot 10^5)}$ is the binary representation of an integer $i$ so that the $i$th bitstring has the value $s$ in the selected indices (as determined by the $x$'s).

You can write CNF clauses that represent the conditions for $x,y$ to represent a valid solution to your problem. In particular:

• For each $k$, exactly one of $x_{1,k},\dots,x_{60,k}$ is true. (This can be expressed in CNF; see Encoding 1-out-of-n constraint for SAT solvers .)

• For each $j,j',k,k'$ with $j\ge j'$, $k<k'$, add the clause $\neg x_{j,k} \lor \neg x_{j',k'}$.

• For each $s,i,k$, add the CNF clause

  $$(y_{s,0} \ne i_0) \lor \cdots \lor (y_{s,\lg(3 \cdot 10^5)} \ne i_{\lg(3 \cdot 10^5)}) \lor \bigvee_j x_{j,k}$$

  where $i_0,\dots,i_{\lg(3 \cdot 10^5)}$ is the binary representation of $i$ and $j$ ranges over all bit indices such that the $j$th bit of the $i$th bitstring is equal to $s_k$. Notice that each term of the form $(y_{s,\ell} \ne i_\ell)$ takes either the form $y_{s,\ell}$ or $\neg y_{s,\ell}$ according to whether $i_\ell$ is 0 or 1, and thus is a known literal in either case, so this does indeed have the form of a CNF clause.

Now run a SAT solver on this problem instance. This will try to find a combination of 8 bit positions that is a valid solution. Of course if you try a solution with 8 bit positions, you can next try to find one with 9 bit positions, etc., and if you don't find a solution with 8 bit positions, you can next try to find one with 7 bit positions, etc., iterating to find the maximum number of achievable bit positions.

For your smaller problem instance, we'll obtain a CNF formula with about $2^8 \times 3 \cdot 10^5 \times 8 \approx $ 600 million clauses and about 632 variables. Each clause will have about 49 literals in it, on average. This is a monstrously large CNF formula, so I don't know whether a SAT solver will have any hope of finding a satisfying assignment for it, but you could try and see.