0

I am trying to use mBART for multilingual translation(about 30 languages) but I am facing an issue with using it as I am currently using langid to identify the languages then load mBART and translate all the words based on the language code that has been identified. But mBART uses this odd format for language codes for example:

en_XX -> English
hi_IN -> Hindi
ro_RO -> Romanian

Whereas Langid outputs them in this format:

af, am, an, ar, as, az, be, bg, bn, br

I cannot seem to find any documentation on how to interpret the mBART language code as even the research paper does not include it.

yudhiesh
  • 213
  • 1
  • 9

1 Answers1

1

It encodes the language and its regional variant, the same way as locales are encoded. hi_IN then means Hindi as spoken in India, en_US would mean American English, en_GB British English. My guess is that en_XX means English in general.

Anyway, the first part of the locale code is the ISO 639-1 language code which is the same as langid uses.

Btw. langid works fine for documents, but not that well for isolated sentences. For isolated sentences, pre-trained FastText classifier delivers much better results.

Jindřich
  • 1,741
  • 5
  • 9
  • Thanks I managed to map a dictionary of fasttext languages codes to mBART language codes with this! – yudhiesh Feb 19 '21 at 10:44