6

I'm looking for a stemmer/lemmatizer for Polish language, preferably in Python. What would you recommend?

I have a list of ingredients in a recipe. Plural forms are inflected differently, depending on the counter, e.g.: for tomatoes

5 pomidorów

2 pomidory

1 pomidor

I want my parser to recognize all those ingredients as for one product, hence the need for stemming.

Community
  • 1
dzieciou
  • 697
  • 1
  • 6
  • 15

3 Answers3

6

There is bunch of lemmatization solutions for polish language. One of the best implementation is in polish morphosyntactic analyser, which you can download here.

It has bindings to python, but you have to install them manually. It is "morphosyntactic analyser" which means, that you get all possible lemmas for a given word. If you want also disambiguation, you can use this tool which is provided as docker container.

My team is currently working on implementing polish language support for Spacy, and we will cover lemmatization, so tool with good integration and support will be available soon.

Gizio
  • 76
  • 1
  • 1
    amazing! What tools are you using for that? Is there any web page about the project? – MkL Oct 30 '18 at 19:46
  • 1
    @MkL We have just started, and we don't have website about, but if you are interested follow spacy pull requests, i think our PR with tokenizer will be ready next week, and lemmatizer in 2 weeks. – Gizio Nov 03 '18 at 17:20
  • @Gizio Will you have any plans to implement full model for polish language in spacy? – Michał Orliński Feb 26 '19 at 10:40
  • Thanks. Do you have plans to include lemma disambiguation into SpaCy? Using docker container is a way to start but does not sound very scallable. – dzieciou Jul 17 '19 at 16:22
2

I have ported Stempel Stemmer from Java to Python: https://github.com/dzieciou/pystempel and improved its effectiveness by pretraining stemming tables on Polimorf dictionary from Morfeusz.

It appears slightly faster than wrapping Java implementation with pyjini but the main point is that I do not have to mix multiple languages in one project, avoiding debugging across Python and Java.

dzieciou
  • 697
  • 1
  • 6
  • 15
1

In PolEval 2017 was a POS Tagging task with Lemmatisation subtask. The winning solution KRNNT obtained about 98% of accuracy in this subtask. KRNNT is available also as a docker image. The results of tagging for your example is (in TSV format):

5   5   num:pl:nom:m3:rec
pomidorów   pomidor subst:pl:gen:m3

2   2   num:pl:acc:m3:congr
pomidory    pomidor subst:pl:nom:m3


1   1   adj:sg:nom:m3:pos
pomidor pomidor subst:sg:nom:m3

As you can see, in the second column the lemma is pomidor (and the third column is POS tag).

djstrong
  • 41
  • 3