2

I am implementing an approximate similarity search using multi-index hashing. I have a set (T) of millions of strings (of same length) and I have a query string(P) (or set of strings) that needs to find similar strings former set. My implementation is the following (assuming length is 64 and similarity is defined as hamming distance less than 15)

  1. Divide each string in T into 16 parts and store each part in a hash table where key is the hash value and value is a vector containing the indices of strings. Ex < 198675, {2, 7, 8, 456}> [I have 16 such hash tables for each portion of the string]

  2. For a query string P, divide it into 16 parts and for each part collect the set of indices(if present) from each hash table and then take the union of all. That becomes a small search space.

  3. Calculate the hamming distance between the query and the resulting set of strings.

I have implemented in C++ and it's working fine, but I am still wondering how could I make it more faster. Any suggestions would be appreciated.

viz12
  • 23
  • 5
  • Would it be correct to assume that in this context Hamming distance means the number of characters that differ (not the number of bits that differ)? – D.W. Apr 13 '17 at 23:36
  • 1
    There's lots of work on the harder version of this question where we use edit distance instead of Hamming distance: see e.g. https://cs.stackexchange.com/q/47794/755, https://cs.stackexchange.com/q/27539/755, https://cs.stackexchange.com/q/2093/755, https://cs.stackexchange.com/q/53299/755. I don't know what is known about the Hamming distance. – D.W. Apr 13 '17 at 23:57

1 Answers1

0

Your method is reasonable. There are other possibilities as well.

Randomized algorithm: locality-sensitive hashing

Here is a randomized algorithm that finds all matches with high probability (probability $1-\epsilon$ where $\epsilon$ is exponentially small, but not zero). It will probably be more efficient.

Build 100 tables. Each table is constructed as follows: Randomly pick a subset of 8 positions (out of the 64 possible positions). Now a string is hashed by keeping only the 8 characters from those 8 positions, and the string is stored in the hashtable keyed on that hash. For instance, if the positions associated with a particular table are $i_1,i_2,\dots,i_8$, then the string $S$ is hashed by hashing $S[i_1],S[i_2],\dots,S[i_8]$ and using that hash value as the key to store $S$ in the table. Each table has a different and randomly generated subset of positions.

Store each of the millions strings in all 100 tables. When you get a query string $Q$, look it up in each of the 100 tables to find all possible matches, take the union of those possible matches, and calculate the Hamming distance between $Q$ and each possible match.

This is very similar to your method, but each "part" is longer (8 characters instead of 4 characters), so there will probably be many fewer "false matches" from the hashtable.

There is no guarantee that this approach will find all valid matches, but most likely it will. The probability that a valid match is not found is

$$(1-(49/64)^8)^{100} \approx 0.0000035,$$

so missing a match is unlikely. You can tune the probability of a missed match by adjusting the number of tables, and you can adjust the parameters ($8,100$) to find the setting that gives you the best possible performance.

This is effectively a form of locality-sensitive hashing (LSH). There may be other schemes worth investigating. You could also look at MinHash.

Trie

Another approach is to store the dictionary of millions of strings in a trie data structure. Given a query string $Q$, you can start traversing the trie, doing a recursive breadth-first search to explore all paths that are at distance at most 15 from $Q$. For many paths, you'll be able to immediately terminate the search.

D.W.
  • 159,275
  • 20
  • 227
  • 470
  • Thanks for the answer. Yes, I have already tried LSH, minhash and simhash techniques. But I found that the execution time is still the same. I am doing some more operations after finding the similar strings, so it seems that portion might be taking more time. BTW LSH based techniques generate some false positives, so I wanted to make sure I am not missing anything. Anyway, thanks again for the answer. – viz12 Apr 14 '17 at 14:55
  • @viz12, it's worth telling us in the question what approaches you've already tried so we don't waste your time (or ours) telling you about something you already know about. Did you try the randomized algorithm I suggested, with many tables? Did you try tuning the parameter choices? If you'd like better answers, I encourage you to edit your question to describe what approaches you tried, why they didn't work, and what the breakdown of running time is with your current approach. – D.W. Apr 14 '17 at 16:34