zlacker

In 2022, most people using NLP use transformers from huggingface. The tokenizer used is written in Rust and used transparently from Python.