Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?by chocrates
- "AI Use Disclosure: A majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history)."
So much for "human programming is obsolete".
by Antibabelic - Nobody except people trying to sell AI are claiming human programming is obsolete though.by Cthulhu_
- For context:
In the final stages of the project, AI was used to assist:
Implementing the user-facing API Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization Porting SIMD strategies between AVX512/AVX2/NEON Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy Refactoring and code reuse
- So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?by swiftcoder
- I'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.by parineum
- the answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.by ProofHouse
- The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.by fastball
- I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OPby 0xnyn
- What sort of setups do people have that are bounded by the speed of the tokenizer?by fwip
- Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
by imperio59 - I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.by avereveard
- It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
by rhdunn - I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.
- If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
by janalsncm - Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!by andersa
- Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
by marcelroed - This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
by zX41ZdbW - Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
by apollopower - Also for embeddings model
- Congrats, I love performance optimizations!
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
by XCSme - > The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)
- The issue is really with testing! You can do such optimizations but you have to be sure that the result is the same in ALL your use cases, so you need very good test coverage and quite good tests as well.
LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.
by lionkor - Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
by ubedan - Is there any write up regarding the SimdJson Algo? Definitely love to read more of it!by a_c
- Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.by marcelroed
- According to Jevons paradox this will likely lead to more electricity used and more CO2 being released to the atmosphere, because it makes it more profitable to build another data center.by o_m
- > I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.
by bob1029 - This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
- For small models, tokenization can reach 1-10% of total inference time.by rockinghigh