4

I have recently ported a stemmer from Java to Python for a highly inflectional language.

The stemmer learns how to change suffixes from the dictionary of words and their inflected forms. It basically builds a stemming table with learned stemming rules. As I was porting the algorithm I decided to train it on a larger dictionary. As a result, the learned stemming table got bigger, and stemming accuracy got higher as well.

Then I thought this actually make no sense as the stemming table size gets closer and closer to the size of the dictionary.

Why build or train stemming algorithms if you can simply lookup a dictionary?

I can understand that in old times storing large files could be a problem, but now? And for some languages there might be no proper dictionary resources. But is there any other reason?

dzieciou
  • 697
  • 1
  • 6
  • 15

1 Answers1

3

There is another reason: words which don't appear in the dictionary. Of course a dictionary approach will correctly stem all the forms which are known in the dictionary, and depending on the language this may indeed lead to better accuracy. However the dictionary approach cannot do anything about unknown words, whereas a generic stemmer can try to apply its generic rules. This can be particularly important with texts which are either very domain-specific (e.g. medicine), which often contain technical words which are not in a general dictionary, or recent user-generated texts such as social media posts where people may use neologisms or words borrowed and sometimes transformed from another language.

Erwan
  • 25,321
  • 3
  • 14
  • 35