TokenDagger
A fast C++ tokenizer with Python bindings that works as a drop-in replacement for OpenAI's Tiktoken, reporting several times faster tokenization in its benchmarks.
Open sourceCoding & Development
Visit website(opens in a new tab)
A fast C++ tokenizer with Python bindings that works as a drop-in replacement for OpenAI's Tiktoken, reporting several times faster tokenization in its benchmarks.