7

This question arose from a practical problem: given a set of texts, find one, which contains a given string (not word).

Let $S$ be a set of $n$ strings, and $l$ the length of the longest string in $S$. What will be the best data structure to efficiently search a string that contains a substring of length $k$ in $S$?

Sequential scan with Knuth–Morris–Pratt algorithm yields $O(n(l+k))$ searching complexity. I thought maybe something similar to tries exists.

D.W.
  • 159,275
  • 20
  • 227
  • 470
Somnium
  • 275
  • 2
  • 10
  • 3
    You should not edit your own question to that extent based on the answers that have already been provided. It changes the requiremets and reduces the value of the answers (which no longer fully answer the question). You should revert the edit and either (1) post a separate question or (2) discuss your point of view in the comments. – Mario Cervera Feb 02 '17 at 12:13
  • 2
    It sounds like after you've read the answers to your original question, you have a new question (can you do better than $O(nl+k)$ running time in a particular case, and where memory consumption must be minimized?). I suggest posting that separately, using the 'Ask Question' button. The current question may be of value to others, so let's not lose that. – D.W. Feb 02 '17 at 17:33
  • @D.W. Ok, simply after getting answers, I inderstood, that I wrongly formulated question. But after thinking more, I thought up an data structure with $O(l(log\ n + log\ l))$ searching algorithm, which is something I hoped for. Now I don't know whether to post new question. – Somnium Feb 02 '17 at 19:23
  • 1
    @Somnium, sure, you are welcome to post a new question and then answer it yourself. Maybe this will be helpful to others. (Just make sure the question can stand well on its own: the problem is clearly stated, the requirements are articulated clearly, the evaluation criteria are explained -- don't skimp on the question just because you know how to answer it. If you allow preprocessing, explain that.) – D.W. Feb 02 '17 at 19:27
  • @D.W. Will ask. And of course I allow preprocessing - I asked for data structure, which itself requires creation, which can be seen as preprocessing. – Somnium Feb 02 '17 at 19:40
  • 1
    http://cs.stackexchange.com/q/35819/755 – D.W. Apr 07 '17 at 21:22

3 Answers3

6

You could append all of the $n$ strings together, and add an arbitrary character '\$' not in the pattern to separate them. Then you could apply the Z algorithm on your original pattern and this new string to yield time complexity $O(nl+k)$ which is more efficient than your suggested $O(nl+nk)$.

Actually, you could still iterate with KMP for $O(nl+k)$ time, beceause the $k$ component is preprocessing the pattern you are trying to find. This only has to be done once, regardless of the number of strings you are checking.

Riley
  • 280
  • 1
  • 11
4

You can use a suffix tree, which is a compressed trie that allows you to store all possible suffixes from the strings in $S$. To search for a substring $q$, you can simply traverse the suffix tree from the root1 to the leaves, making comparisons with the edge keys (each of which represents one or more characters). If a branch does not exist in the tree, then $q$ is not a substring in $S$; otherwise, it is.

If you want to find out the exact string in $S$ that contains $q$, you can annotate the nodes of the suffix tree so that they contain this information (e.g., you can add a string number and a starting position). This is the idea generalized suffix trees are built upon.

The running time of this approach is $O(n \: l)$ for the construction of the suffix tree, plus $O(k)$ for the substring search, for a total of $O(n \:l + k)$.

1 Note that every substring of a string $T$ is the prefix of exactly one suffix of $T$. Therefore, substring search can always be performed starting at the root of a suffix tree.

Mario Cervera
  • 3,704
  • 2
  • 19
  • 22
  • I don't fully understand, using suffix tree and searching starting from its root traversing $k$ nodes, I can find which string from $S$ contains given string in $O(k) time$ and preprocessing being $O(n l)$? What will be space complexity? – Somnium Feb 02 '17 at 20:06
  • 1
    Yes. You need $O(nl)$ time to build the suffix tree. This is because $nl$ is an upper bound on the number of characters in $S$ and building a suffix tree for any arbitrary string $T$ takes linear time in the length of $T$. Once the suffix tree is built, you can search a substring $q$ of length $k$ in $O(k)$ steps. At each step, you go down one level of the tree, matching the next character of $q$ with one of the child pointers of the current node. If, at some point, you cannot continue, then $q$ is not contained in any suffix, and, therefore, it is not contained in $S$ (cont) – Mario Cervera Feb 02 '17 at 20:35
  • 1
    If you can fully process $q$ and stay within the tree, then $q$ is definitely a substring of (at least) one string of $S$. To find out which one, you need additional information in the nodes of the tree. Have a look at generalized suffix trees to find out how this can be accomplished. – Mario Cervera Feb 02 '17 at 20:39
  • 1
    The space complexity of a suffix tree for a string of length $n$ is $O(n^2)$ because the average length of the $n$ suffixes is $n/2$. However, linear space is sufficient if we compress the edges. Compressed edges do not represent individual characters, but, rather, substrings. – Mario Cervera Feb 02 '17 at 20:53
  • However, generalized suffix tree (as described on Wikipedia) is using compressed edges, which then has optimal time and space complexity. Thanks! – Somnium Feb 02 '17 at 21:14
3

The answer depends on the number of patterns, the total size of the strings $nl$, and how many times you expect the pattern to occur in each string.

  • If you have a few patterns, a standard string matching algorithm such as KMP is sufficient.
  • If you have many patterns at the same time, you should use an algorithm that preprocesses the patterns (e.g. Aho-Corasick).
  • If you have even more patterns or the patterns come in multiple batches, you should preprocess the strings instead. The suffix tree is nice in theory, but it's slow and memory-hungry in practice. The suffix array works better, and the FM-index is a nice substitute if you don't have enough memory for the suffix array.
  • If you have really many patterns in a single batch, you should preprocess both the strings and the patterns and match the resulting structures against each other. The basic idea is straightforward, but you'll have to do some work to get the details right.
  • If you preprocess the strings and expect the patterns to have many occurrences in each string they match, you should augment the index (e.g. suffix array or FM-index) with a document retrieval structure.
  • If you preprocess the strings and even the FM-index won't fit into memory, the string B-tree should work reasonably well.
Jouni Sirén
  • 462
  • 2
  • 5