Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Useful but how it compares with other providers model
- Thanks. I'm not sure if you're asking "how does frugon compare provider models" or "can it compare across provider models" or even "how does frugon compare to other tools"...
On the first: It does this utilizing pricing from LiteLLM registry, and quality tiers from LMArena.
On the second: A log of calls for OpenAI can get recommendations for Gemini or Anthropic or DeepSeek etc. and vice versa.
On the third: It's free, local and offline. Hosted tools normally require you to plug them into your live traffic, whereas frugon reads logs you already have. And you have the benefit of exploring the source code.
by jarodrh - Haha. somehow i just love the naming. it just makes sense :Dby XUEYANZ
- Thanks! I love a good meaningful name
Frugal + on
:P
by jarodrh - this would be more interesting as a local LLM anlysis; throw out all the costs, and figure out primary-subagent model architecture, and maximize token generation and prefill.
I don't see how anyone can operationalize this information.
by cyanydeez - That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a cost perspective as opposed to a raw token generation/prefill perspective. The mindset can be applied to both sides, but Frugon prices the cloud side.
The end goal is essentially the same and your mind went to that point, "primary-subagent model architecture". This is what the tool helps you figure out. It's not there to hold your hand and explain what your architecture should be, as I wanted it to be a small simple tool that would give the user insight into triggering your exact thought process. The thought process of breaking down their tasks by type, mapping that to individual models regardless of app or dev tool, regardless of cloud or local (the thought process transfers). It shows the user that they could route a portion of their calls to a cheaper model. It's then up to the user to understand the task type and point those calls to different models.
To directly answer the operationalization statement, it depends on what the call log is from (app/harness).
If harness, then the direction would be to pin models per role per task type; the routing recommendation maps directly to that, and measure/judge allows you to verify this before switching.
If app, then this is a similar shape as above, where you would then categorise and pin those calls identified in the recommendation to the model recommended, and as above, measure/judge allows you to verify before switching.
You're proposing an optimisation for throughput whereas Frugon is for spend. Same issue, just different lens.
by jarodrh - A cheaper model can look fine per request, but one weak answer may create another call or a human review step. That seems like the hardest cost to capture.by mujia
- True, but human review isn't visible in the logs, hmm, however I suppose retry patterns can, but frugon doesn't currently track this, so we can't quantify this yet.
Frugon analyzes your existing logs offline on cost and quality tier. The "looks fine per request" is exactly the reason why --judge exists.
Earlier, cyanydeez inspired the consideration of a metric "effective cost per judged success", which could also answer this point.
Try it on your logs, and tell me if anything falls through the cracks.
by jarodrh - I think this is great, and the next frontier is to analyze how well calls can be handled by a local model. Realistically, to do that usefully requires response time as a new dimension of judging: Can a local model provide an acceptably accurate response in an acceptable amount of time?by pauldavis
- Thanks. Yes, local models are gaining a lot of traction.
The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.
Noted: "response time as a new dimension of judging" - Added to the roadmap.
Try it and let me know if you hit a wall.
by jarodrh - This looks super clean. I'm curious about the --judge command. How does it evaluate if the cheaper model's response is a "tie" or acceptable? Is it using a specific LLM-as-a-judge prompt template?by isadubois
- Thanks. Yes, that's exactly right, and aptly named so in the code.
A few things to note: the prompt is actually deliberately tie-biased; the tie only then breaks on clear material differences: factual error, missing information the prompt requested, or just a complete failure to follow the instruction. The judge is explicitly told that response/output length, wording, style and formatting are not quality differences.
To ensure there's no favouritism, the judge only sees the outputs (A and B), it never sees whether the output is from the current model or from the candidate; and to ensure no funny business occurs, the outputs for A and B are randomized.
The default model is chosen automatically from the highest-tier model in the log, but this could invite self-bias (this is cautioned in the report output), so you can override this with --judge-model
You can check out the prompt for yourself in src/frugon/measure.py - search: JUDGE_PROMPT_TEMPLATE
by jarodrh - Evals and OpenInference (OpenTelemetry) might be useful.
Costed opcodes (like the shelved eWASM opcodes cost chart) would be useful for this model routing problem as well.
Is this the cost to converge problem, the minimize cost to converge upon sufficiently low error problem, or the minimize cost and error problem?
EA methods: mutation, crossover, selection
Gradient descent as a mutation, crossover, and selection pattern; back up when the error/cost stops decreasing for too long and try a different branch.
A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.
by westurner - Very interesting thoughts. This is a whole discussion on its own.
The ones that I mulled over that I feel have legs in my line of thinking are:
> minimize cost to converge upon sufficiently low error problem
- This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs.
> Evals and OpenInference (OpenTelemetry) might be useful.
- Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference.
> A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.
- This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.
by jarodrh - What we need is an AI gateway/router that will actually first analyze the input tokens and then decide what model to use. If it's so simple that a dirt cheap qwen 3.5 flash or whatever will be fine, then it chooses that. If it deems we need GPT 5.6, then it uses that, etc. does anything like this already exist?by indigodaddy
- Yes they do, it's quite the popular topic atm. RouteLLM (OSS - frugon's savings bands are based on their research), OpenRouter's auto mode; I actually commented on a /show post not long ago: Wayfinder Router (neat project) - https://news.ycombinator.com/item?id=48704373 and more.
My stance on the router point is a config-led (deterministic) one, even if the eventual industry consensus is a hybrid of dynamic (AI-led) + deterministic. Both sides still require evals/evidence for policy creation, which is what frugon provides.
by jarodrh