Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Deja-vù: 15 years ago I was working for AWS, and at some point (2010-2011) tens of startups came out to provide cost optimization, cloud switching, etc.
If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.
- We built something similar internally after AI spend on support-ticket triage kept beating estimates. Routing itself wasn't the hard part, it was that our dashboard only tracked immediate task completion, and had no idea a cheaper model had quietly produced a worse answer that the customer reopened a week later. We ended up tagging every response with which model handled it and joining that against reopen rate a month out, and the gap on the cheap tier was bigger than we expected going in. Curious if Tokenless has thought about exposing a delayed quality signal like that, not just pass/fail at the time of the call.by jartan2002
- How is this different from OpenRouter?
I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average:
https://github.com/rush86999/atom/blob/main/docs/architectur...
by rush86999 - There is no difference.
It is just another metoo copy of OpenRouter, but "Backed by Y Combinator™"
by rvz - Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)
Sounds like DigitalOceans Model synthesis (https://www.digitalocean.com/blog/model-synthesis) or AilinOne (https://ailin.guide/) ?
by crazytweek - Sounds like this would be expensive on the input token side with some saving on the decode side. For workloads with large documents / context that could be an issue.by sarjann
- These approaches, along with OpenRouter Fusion (https://openrouter.ai/blog/announcements/fusion-beats-fronti...) are different from ours in that:
1. we trained custom models to predict LLM performance (this performs much better than asking LLMs directly to predict performance)
2. we cut off model responses early if they are not performing well, saving on costs versus running them to completion
3. finally we serve the unmodified output of the chosen model, instead of synthesizing the outputs into one, which makes us a model router
by random17 - It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.
τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.
Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?
by MikhailTal - Thanks for the praise!
We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.
Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)
We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).
by rohaga - > Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track
By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.
by JoshTriplett - Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models.
We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made
by rohaga - I do wonder why not just use cheaper models like Luna or Grok instead of dealing with routing. DeepSWE has Grok at 54% for $2.42 or 67% for $3.03 for Luna. Tokenless Pro is 6.67 for the same performance as Luna.by siliconc0w
- Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.
One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?
Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.
by popPopBoom - Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.
We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.
by rohaga - The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?
- I ran some tests last quarter with a similar multi-model routing setup focused on code completion tasks. Using an open-source Claude alternative combined with Claude-instruct for fallback, I saw about a 45% cost reduction compared to consistently hitting the Claude Code API alone, with latency penalties under 300ms for the fallback requests. The key was tuning the confidence threshold so that around 70% of calls got handled by the cheaper local models without sacrificing accuracy much.
One interesting data point was that for smaller completions (~50 tokens or less), open-source models like OpenHands outperformed cloud-hosted Claude alternatives in terms of 95th percentile latency—likely because no network involved. But for bigger, multi-file refactoring queries, fallback to frontier models was unavoidable to hit the 90+% pass rate. The dynamic routing balanced those two extremes to keep API spend down while still scaling.
That said, the caching layer remains a bottleneck. Changes in the input context or tool states quickly cold the cache, forcing more queries to the expensive model. For workflows with lots of sequential tool calls (e.g., automated code lint + formatting + test generation), the raw switch cost was around 15-20% overhead compared to fixed-model baselines until cache warm-up happened. This matches the experiences others have reported.
- Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).
I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.
Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?
- We have run it on DeepSWE, check out our blog post for details of how we leverage the intelligence of the smart models https://usetokenless.com/blog/building-tokenless/by rohaga
- So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.
But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.
How do you deal with that?
by mediaman - Pretty sure this is a start-up not an end-up. This is exactly what they will need to iterate on to drive costs down. And they’ll be in the perfect place to do it because they’ll have the telemetry. Exciting!
- > So this only switches models if the cache is cold
Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.
Remember that Deepseek can be something like 30x cheaper even cold.
There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.
by rohaga