Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I see a great tension in the market today. On one hand you want agents to work reliably and that needs a lot of harness, computer use, model routine, tasks running longer etc. And on other hand you simply want to reduce your dependencies and costs. Agent building is very nascent and all the frontier companies are trying to build the best harnesses possible. As basic prompting, researching, coding gets mature, more and more of such tasks will be optimal for model routing open source etc etc but there is a chance that by that time frontier models again make costs and routing, low and effortless. Basically I believe everyone has started jumping to the. -- REAL PROBLEM IS COST v. REAL PROBLEM IS EFFICIENT, RELIABLE AGENTS/WORKFLOWS. It's going to be very interesting to see how this plays out.
    by implexa_founder
  • I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.

    I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .

    by peterbell_nyc
  • This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.

    Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.

    For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains

    by GodelNumbering
  • Looks interesting!

    Out of curiosity, how does it compare with vLLM Semantic Router?

    For reference:

    https://vllm-semantic-router.com/

    https://github.com/vllm-project/semantic-router

    vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444

    https://github.com/vllm-project/semantic-router

    For instance, does it offer similar algorithms:

    - vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/ - vllm-sr/fusion: panel-style multi-model reasoning and synthesis. - vllm-sr/flow: router-native workflow orchestration - vllm-sr/remom: multi-round reasoning over one or multiple models.

    FWIW, it does look good on https://routeworks.github.io/leaderboard

    Ref.

    RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers, https://arxiv.org/abs/2510.00202, https://github.com/RouteWorks/RouterArena

    by matt_d
  • Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
    by g00k
  • It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.

    It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.

    by jakozaur
  • The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.

    How does this router translate to $$$ when developing?

    by stpedgwdgfhgdd
  • I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback:

    1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up

    2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ultra, implementation to mid / high etc. They know when they're exploring, planning, implementing, reviewing etc. and which model class to select and when it fails.

    With a proxy you're breaking this control loop and feedback. It doesn't know, for ex. that it just attempted with deepseek v4 and it failed, lets try Opus?

    3. How are you going to RL improvements and prevent the router becoming stale? You only have access to your own internal prompts and ~thousands of samples.

    This is RL'd on one orgs codebase. There are going to be a lot of prompts you haven't seen before and have no insight to on how to route correctly, and you have no insight into users HF to improve your own model. Orgs aren't going to share their traces with you, so you need other sources to train on and improve

    There are also new model releases every week that you need to keep up with - whats the story going to be here

    4. Publish evals by running terminalbench / deepswe bench. Show us the performance / cost / time chart vs the other agent and model sets. If you can show gains there, you have a very simple value prop to sell where you can charge for a % of the saved costs

    by nikcub

Explore Birbla archives