Comments
Hacker News
What sort of setups do people have that are bounded by the speed of the tokenizer?
by fwip
This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
by zX41ZdbW
Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
by apollopower
Congrats, I love performance optimizations!
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
by XCSme
Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
by ubedan
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.
And screw all the 0.1% haters on here, this is great stuff.
by cschmidt
Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
by maxdo
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What sort of setups do people have that are bounded by the speed of the tokenizer?by fwip
- This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
by zX41ZdbW - Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
by apollopower - Congrats, I love performance optimizations!
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
by XCSme - Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
by ubedan - This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
- Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.
And screw all the 0.1% haters on here, this is great stuff.
by cschmidt - Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
by maxdo