Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I notice Cursor already does something similar. Even if I have Opus 4.8 selected, it will trigger subagents using Composer 2.5. I like using Auto personally because it is pretty effective and deeply discounted, but at work I YOLO Opus high.
I imagine a solution like this will eventually be an enterprise-forced solution because there is no reason right now for individual developers to be selective about model pricing. Even more important is non-tech users who do stuff through MCPs like "give me a full overview of all analytics" and let it chug for half an hour.
by lubujackson - Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.by adchurch
- Your employees need to stop baiting on X.. "I'll hire cracked devs who score above a certain number, just scan your GitHub, get a score, send me a screenshot" meanwhile getting the actual score is a premium feature.
This is one of the most grotesque metric and funnels for data input I've seen.
And, to delete your Weave account? Email them, they don't respond.
by TaylorGood - There are so many of these projects to wrangle AIs I think we might need an AI to go through, analysing each and amalgamating the good bits.
It makes me think of MakeFiles.
Make is sufficiently bad that everyone who has used it has considered writing a better way to do it. A good percentage of those people have done so.
On the other hand, make is also not so sufficiently bad that it cannot do its job. The choice becomes picking the thing that everyone has or one of the many many alternatives that proclaim their strengths and leave their weaknesses lurking to bite when they are least expected.
No single replacement to make dominates, and make lives on. I wonder if AI management is on a similar path.
by Lerc - I ran into a problem at work recently: we are given access to a bunch of models up to a full Claude Opus 4.8, but a monthly budget of 100k tokens. We are also given access to Gemini 3.5 Flash & 3.1 Pro with a daily budget of 50M tokens, but no tool calling. I'd love to hook Claude Code (or Pi) into the Gemini model, but the lack of tool-calling makes it quite difficult. I've been planning out how an intelligent router might be able to use a token-efficient tool-calling model (including a small local open-weights model) to handle the basic tools like reading from the file system or interfacing with MCP servers such that context is gathered, but then send the built up context to the Gemini model where I have a nearly unlimited (for my use cases) token budget.
Could your router handle this?
by elgertam - Monthly budget of 100k Opus tokens? So $2.50 worth?by Jaxkr
- Yes we can route to Gemini models too and we handle all the translation complexity there!by adchurch
- I’m curious how a workplace ends up with a model policy like this. It seems like you’d spend more time trying to work out how to use a tiny number of Opus tokens than doing it yourself.by thombles
- I see a great tension in the market today. On one hand you want agents to work reliably and that needs a lot of harness, computer use, model routine, tasks running longer etc. And on other hand you simply want to reduce your dependencies and costs. Agent building is very nascent and all the frontier companies are trying to build the best harnesses possible. As basic prompting, researching, coding gets mature, more and more of such tasks will be optimal for model routing open source etc etc but there is a chance that by that time frontier models again make costs and routing, low and effortless. Basically I believe everyone has started jumping to the. -- REAL PROBLEM IS COST v. REAL PROBLEM IS EFFICIENT, RELIABLE AGENTS/WORKFLOWS. It's going to be very interesting to see how this plays out.
- I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
- "Based on your subscription tier and local hardware here's a list of models that fit and process definitions your biggest brain will comfortably handle."
I guess that sounds a lot like moving your evals and auto tunes to a third-party, but I don't have the time, budget, or inclination to create a system like this out of whole cloth and then keep it relevant.
I could see something that provides on-the-fly routing information being useful, but actual decision-making is too dependent on context.
by gopher_space - This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.
Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.
For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains
- When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively:
1. Small models can carry out a good number of requests e2e 2. Small model for part of a request + cache miss < big model for entire request in many cases 3. Subagents
For our own usage we've saved 40% so far (that is of course including costs of uncached requests when switching models)
by adchurch - Looks interesting!
Out of curiosity, how does it compare with vLLM Semantic Router?
For reference:
https://vllm-semantic-router.com/
https://github.com/vllm-project/semantic-router
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444
https://github.com/vllm-project/semantic-router
For instance, does it offer similar algorithms:
- vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/ - vllm-sr/fusion: panel-style multi-model reasoning and synthesis. - vllm-sr/flow: router-native workflow orchestration - vllm-sr/remom: multi-round reasoning over one or multiple models.
FWIW, it does look good on https://routeworks.github.io/leaderboard
Ref.
RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers, https://arxiv.org/abs/2510.00202, https://github.com/RouteWorks/RouterArena
by matt_d - I tried Sakana Fugu, boy is it hungry ... it blows up tokens like nothing I have ever seen. Not that impressed with the results I got from it however if I'm being honest. Now I'm bought into their buy 1 get 2nd month free so will keep trying it but may cancel after.by hmokiguess
- Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).
As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.
by adchurch - Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.by g00k
- Yep this was always the reason to avoid "auto" mode in cursor.by alansaber
- > Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using.
Perhaps you're just not the best use case. It may work better when Average Joe is the one prompting.
- Yeah that's a really interesting point, tbh I think the more relevant variable here is the harness you're using rather than the specific model? i.e. GPT 5.5 in the Claude harness behaves a lot more like Claude than Codex if that makes sense.
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
by adchurch - It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.
It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
by jakozaur - Yep cache awareness is super important, mentioned this in another thread here: (https://news.ycombinator.com/item?id=48689448)
But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience
by adchurch - The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.
How does this router translate to $$$ when developing?
- Artefact-based workflows solve this problem, and I think it’s more effective to go in that direction.
I still have Claude Code because Opus makes good plans, but I hand the plan over to M3 on Pi with 99.9% cache hits on a long session. Lovely. Pi then makes a summary file that Opus can use to review the code/context.
But you do need them to write down their stuff, so that compaction and clear sessions can work off a nice, concise document.
And if you are simply using Claude Code, then /advisor is what you want: a sub-agent with a much cleaner context is spawned to handle something -> not cached per se, but much cheaper to run.
I’d stay away from workflows that automatically route between models unless you can afford the cache misses. That’s also why GLM 5.x is costing me much more, I don’t get good caching with it.
by port11 - You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
by adchurch - I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback:
1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up
2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ultra, implementation to mid / high etc. They know when they're exploring, planning, implementing, reviewing etc. and which model class to select and when it fails.
With a proxy you're breaking this control loop and feedback. It doesn't know, for ex. that it just attempted with deepseek v4 and it failed, lets try Opus?
3. How are you going to RL improvements and prevent the router becoming stale? You only have access to your own internal prompts and ~thousands of samples.
This is RL'd on one orgs codebase. There are going to be a lot of prompts you haven't seen before and have no insight to on how to route correctly, and you have no insight into users HF to improve your own model. Orgs aren't going to share their traces with you, so you need other sources to train on and improve
There are also new model releases every week that you need to keep up with - whats the story going to be here
4. Publish evals by running terminalbench / deepswe bench. Show us the performance / cost / time chart vs the other agent and model sets. If you can show gains there, you have a very simple value prop to sell where you can charge for a % of the saved costs
by nikcub - Really appreciate the thoughtful feedback!
1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're aware of that when making routing decisions
2. The agents are model aware yes but they are not incentivized to optimize too heavily here (in particular they don't use OS models even when they would be better). I think that's where this router comes in and brings genuine improvement.
3. Two parts here: 1 is continuing to grow our golden dataset over time, 2 is using reward signals from production traffic (on a per-customer basis or, if allowed, across all users)
4. Yes we have these internally, great callout that we should publish! Will do + will link from the repo soon. (Fwiw I think these benchmarks are useful but don't fully capture vibes - you should try it out yourself for that!)
by adchurch