2

ReDos attacks exploit characteristics of some (otherwise useful) regular expressions ... essentially causing an explosion of possible paths through the graph defined by the NFA.

So does using a recursive-descent parser (such as ANTLR) necessarily avoid the problem?

David Bullock
  • 189
  • You're going to fire up antlr when you want to see if the email address appears to be valid? –  Sep 04 '15 at 02:17
  • Perhaps that would be a good idea. http://emailregex.com – Alan Shutko Sep 04 '15 at 02:48
  • If firing up ANTLR would make my code immune to a ReDoS when parsing an email address, that's something I would weigh up when when I don't control the inputs. Am open to other ways to avoid ReDos. – David Bullock Sep 04 '15 at 03:13
  • @DavidBullock don't write code (be it a regex, cfg, or java or etc...) that allows you to get into exponential sized loops or recursion. –  Sep 04 '15 at 03:18
  • @MichaelT nice advice, but does it follow that for every evil regex which validates sentences for some input grammar, that there is a non-evil regex which performs the same validation? I would hazard not. So would a different tool (recursive-descent? other?) do better? Or is the evilness 'in the grammar' regardless of the tool? – David Bullock Sep 04 '15 at 03:38
  • @DavidBullock that would be an interesting question to ask some computer scientists. I'm an industry programmer and tend to be a bit more pragmatic. If the language is regular, I use a regular expression. If the language isn't regular, I say "screw it" and accept it without validation. Spinning up a LR matcher is overkill. –  Sep 04 '15 at 03:47
  • @MichaelT I asked. Whether an LR matcher is overkill or not depends on the security context, yes? And it's useful to know and understand trade-offs, right? My question seems pragmatic to me. – David Bullock Sep 04 '15 at 05:24
  • If you can write if these errors, it doesn't matter. If you can't, or don't know how to identify them, using a cfg will likely just make it harder for you to verify if that has errors. –  Sep 04 '15 at 05:42
  • 1
    You do realize that the "evil" regex is something that you create, right? – kdgregory Sep 04 '15 at 10:58

2 Answers2

2

Necessarily? No.

Recursive decent parsers can be implemented just as naively as the regular expression algorithm (you could implement the NFA via recursive decent for example). They're arguably more likely to have the problem because they usually have to deal with context free grammars, which allow a wider range of complexity and thus a wider range of possible exploitable use.

Telastyn
  • 109,398
  • As I understand it, the 'path-explosion' problem with the NFA is due in part to the fact that it can only consider the current state and the current position in the string. The number of possible paths increases exponentially with regard to the length of the input string. Wouldn't the top-down-ness of a recursive-descent parser permit fewer paths and therefore preculde that particular problem? – David Bullock Sep 04 '15 at 03:30
  • @DavidBullock It... depends. You may end up with a better memory characteristic, but similar run-time characteristics, especially if you want to consider all possible matches. – Vatine Sep 04 '15 at 10:53
  • @DavidBullock - it depends on how much look ahead the parser uses. Just because recursive decent can use more look ahead doesn't mean it does. Most generators try to do LR(1) or LALR(1), meaning only one character of look ahead - just like Regexes doing NFA style work. – Telastyn Sep 04 '15 at 11:45
1

Each regular expression defines a regular language. A regular expression can be translated to a non-deterministic finite automaton (NFA) that recognizes its language. Each NFA can be translated to an equivalent deterministic finite automaton (DFA) that recognizes the same language in linear time. Each DFA can be translated to a regular grammar (see e.g. here) that generates the same regular language. A regular grammar is a particular type of context-free grammar, which implies that all regular languages are also context free.

So, if the language you want to recognize is regular, you could use a recursive-descent parser to parse it, but you would not gain anything in terms of time-complexity.

As far as I understand, the attacks you mention in your question are related to

  1. Naive implementations of the parsing algorithm that are not O(n).
  2. Extensions to regular expressions such as back-references, which make the resulting language non-regular and force one to use some ad-hoc algorithm that has an exponential worse-case complexity.

Case 1 is easily solved by using a non-naive implementation that runs in linear time: no need for recursive-descent parsers.

Case 2 depends on the extensions you want to support. If you want to use recursive descent parser your language must be context-free, so you must first check if languages defined by regular expressions with extensions like back references are context free.

These languages are not context free (see the answer to the parallel question on the Computer Science site for very good background on the topic). My intuition as to why they are not context free is that a context free grammar would need to be able to generate strings with arbitrarily long disjoint matching substrings. You can probably use the pumping lemma for context free languages to show that a context free grammar cannot do this.

Bottom line: recursive descent parsers are no use for this kind of extensions.

A pragmatic solution is to first run the fast DFA algorithm and only revert to the potentially exponential algorithm if a back-reference is encountered in the regular expression. This is mentioned here in Section Implementations and running times. Unfortunately there is very little information about this approach in the wikipedia page. If I find more details somewhere else I will add them to this answer.

Giorgio
  • 19,646
  • In general it may be true that the class of extended regex we're dealing with is not context free, but in reality many (most, I would guess) examples in the wild of actual applications of this facility are written to process languages whose grammars are context free, e.g. email addresses, whose syntax is defined in BNF in RFC2821. Recursive decent parsers are clearly able to parse email addresses, and probably most other real applications (the other popular example of the technique is matching strings of balanced parentheses, which is also a valid context free grammar). – Jules Sep 04 '15 at 18:44
  • @Jules: "In general it may be true that the class of extended regex we're dealing with is not context free...": I am not sure I understand your use of in general. Either all the languages defined by these regexes are context free or some are not. If some are not, then you cannot fully support regexes with evil extensions using only recursive descent parsing. If context free languages are sufficient for a certain application (e.g. email addresses), then you can use another definition of regex without evil extensions for which recursive descent parsing is sufficient. – Giorgio Sep 04 '15 at 20:06