Building BERT tokenizer with custom data

Question

I'm wondering if there is a way to train our own Bert tokenizer instead of using pre-trained tokenizer provided by huggingface?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. — Community, Jun 20 '23 at 13:43

score 1 · Accepted Answer · answered Jun 20 '23 at 14:22

1

Yes, of course, it is possible, but that implies that you need to train again the whole model, not only the tokenizer.

Here you can find the steps to do everything with the HuggingFace libraries, including a complete example.

answered Jun 20 '23 at 14:22

noe

1 Answers1