Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What was the architecture of your router? If it was based on GRPO/RL, it would be interesting to hear why your router performance capped.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.
by atlex2 - We tried different things: heuristic, TF-IDF and LLMs. I voluntarily didn't talk about the tech because it doesn't fix the problems mentioned by switching models on the fly.by brunaxLorax
- Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers.
The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.
E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.
by scionaura - 100% agree. MoE is the perfect example: reducing FLOPs while keeping intelligenceby brunaxLorax
- This arbitrage isn't durable either though, as it's just a race to the bottom with open research, and the long term destination of everything is custom models for different tasks, since it's going to become increasingly apparent that some areas of knowledge have anti-synergy as we push up the diminishing return curve.by CuriouslyC
- The 'personality' of the model matters a lot even for stuff like Code Gen. If you're building something with Fable and then get routed to Opus cause the router decided your task is not 'demanding enough' it's like: great, now I have to read complex prose like "That's the thing, and the thing is the point:..." Opus-isms when I just wanted to make a shopping cartby firasd
- Agreeby brunaxLorax
- It seems apparent to me that task complexity can’t be determined by prompt alone. How? A prompt is just a simple rambling. An agent will go through many many tool calls and steering just to arrive at the right approach.
A serious router therefore needs to build up a dataset of how different models responded, end to end, to different tasks on different contexts. I won’t comment on whether the current frontier models can reliably judge these outputs, but I am sceptical about that. And moreover look at the state of evils! They are gamed to hell and keep losing credibility.
A more difficult problem for router builders is that they are working on an opaque system behind an external API. How can you reliably guarantee model behaviour when model behaviour has been shown to deteriorate under arbitrary conditions that have nothing to do with the task given? So much investment only to be an AI company that can get rug pulled by the real AI companies at any given time.
I would think the only people who can come up with good routers for a collection of models are the inference providers themselves. Because theoretically they have full control of how their models are served. And inference is not zero cost or cheap for them either. And going by OpenAI’s experience routing is not an easy problem for them to solve either. And they don’t have the incentive to route you to cheaper models and reduce costs for customers at the same time. Routing objective for them is to increase their own profits.
by kaycey2022 - I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
by mmargenot - > Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
by randomblock1 - Sounds like an orchestrator for me ratter than a routerby brunaxLorax
- I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.by bluejay2387
- I recently wrote about first principles of model routing that I've learnt building a model router.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
by try-working - I agree with one distinction - coding agent workflows can use defined subagent roles that are pinned to specific models and I have found this very effective. The orchestrator is building all the context to make these assignments - it’s not a dumb router. Using Minimax M3 for exploration and librarian tasks for example is fast and cheap - my $10 plan lasts all month and saves a lot of tokens for my main coding plan.by jeremyjh
- Yes. orchestrator > smart routerby brunaxLorax
- Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
by velcrovan - what’s wrong with the sentence? reads fine to meby hhh
- a French human doing its best to sound punchy in English :)by brunaxLorax
- I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
by dweez - I mean can't you just directly test some variations of the query against many models at once and pick the cheapest model that hits your accuracy goal? If it's a one time run then ofc this is all pointless, but for ongoing tasks it makes sense. This seems more logical to me than making another AI model of some sorts intuit the right LLM for the job.by JarJarBeatU
- How does one spend a lot of time researching model routing? Did you evaluate different routing providers? Maybe you could build a rorouterter that lets you easily switch to any router, ideally if there's a vuln in any of them you can get infected even if you don't use it.by TZubiri
- You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
by dofm - I think routing should be pushed 'down the stack' so to speak.
What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.
But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.
This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.
by all2 - > Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
by overgard - I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.by zackify
- It's hard to keep track of all those launches, I agree. I would separate 2 usages of LLMs: - LLM is integrated into a workflow/app for a classification or whatever job: scoring outputs (evals, KPIs) and choosing a fit between quality and cost is enough - LLM is used as a tool you interact with (like coding or research): here is where you have to know your brushes. As humans we get used very quickly by the model personality, the format of output, the way it "thinks" and interacts.by brunaxLorax
- I just ask our fleet maintenance agent to configure the variants of the new model and then run the bakeoff. In a few hours, I get a doc containing exactly how it performed on my real tasks, against the other models we've benchmarked.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.
by daeken - I'm all for dropping priority on vendor neutrality and tightly coupling with vendors.
But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.
I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.
by TZubiri - You do still need to know though, different prompts have different affects on them.
- A programmer is someone who declares "I understand the problem so well that I can teach a computer to solve the problem, even for examples I have never seen before".
If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.
by mrkeen