Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!by bambax
- As long as you are constantly diligent in staying on top of what each provider is doing that may impact you.
It reads to me an argument for self hosting, maybe a less capable model to deal with smaller compute resources, but when starting to use that model and inference software to build a benchmark you can easily rerun when you change things to observe the impact of the change.
Maybe that’s a similar level of diligence but they feel different to me.
by bradfa - I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times.
The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels like it oscillates between "I can't believe it's not Sonnet", but it costs a fraction of that and "This feels like I'm back using GPT-4, why am I even bothering with an LLM?".
by Macha - Using the same provider?by copperx
- LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder.
So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!
There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do. Even AWS Bedrock has a Claude that feels different to me, but I haven't had a chance to do actual benchmarks that would show that.
by epistasis - Bedrock Claude is absolutely not identical to 1P Claude.
They’re close enough to not matter though.
by dannyw - That looks absolutely horrifying. What are the alternatives ??by bsaul
- Vercel AI Gateway is one I've used: https://vercel.com/ai-gateway/modelsby james-bcn
- I've used Requesty (https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.by danvdb
- Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with.
It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.
by JaceComix - I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.by dools
- I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/by vinhnx
- Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.by maeln
- I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.by john01dav
- There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
by bakugo - Pretty damning. We're using OpenRouter for some research tasks and it's making me question everything. I suspect lots of the model providers are running into issues like these: https://forum.level1techs.com/t/why-your-local-llm-feels-dum... (HN discussion: https://news.ycombinator.com/item?id=49402232)by binarymax
- If you require consistency for research, I wouldn't recommend relying on hosted inference without validating the provider/inference stack very carefully.
For my contribution to OpenAI's gpt-oss red-teaming competition (published as arXiv:2510.01259), I initially used OpenRouter with DeepInfra and Together AI. The results were much too noisy to draw reliable conclusions from, and markedly different behavior through AWS Bedrock was the last straw. I ended up renting GPUs through vast.ai and running the model myself with vLLM.
by ndr_ - Co-founder and COO of OpenRouter here.
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
by numlocked - I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket.
To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.
by noahbp - I switched to Opencode because they have guaranteed performance. Maybe you could have an openrouter certified badge or label for this sort of thing.by Computer0
- Thanks for the insight here.
One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close.
This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase the first part actually uses their own customized inference program rather than the standard methods that all the third party providers use. This isn't nefarious, it's just the challenge of these sorts of stochastic systems.
Having been in science for decades now, and seen benchmarking across many different fields, these results are completely expected for me. LLM serving is not mechanical, it's hard to get right and has lots of unknown footguns. Even something as extreme as scrambling a matrix will still likely get results that are nearly as good as normal, and if there's a bug deep in vLLM or the tensors metadata that results in that, then it's going to be pretty hard to find unless you're an active researcher with knowledge of the particular model you're running inference on. I kind of doubt that's happening here, but maybe!
In the scientific literature, when benchmarking methods, everybody's own method performs best in their own hands. Some attribute it to researchers gaming benchmarking for publication purposes, but I think it's just what we see here: the people who made a method are just the best at using it because they know all the quirks and use it best.
Programmers are not used to thinking with that nuance, and jump to conclusions about lying about quantizations, etc., but this is really just an unavoidable part of AI/ML methods: when things aren't perfect they're still pretty good and it's going to take the model creator to truly debug it. At least until the open weights ecosystem gets a lot better at ensuring reproducibility, and model cards are nowhere detailed enough for that to happen yet.
by epistasis - Thanks for the response Chris, I couldn't make my business work without OpenRouter in the first place, so kudos
This was meant as more of a technical reference, sorry you had to wake up to a PR drill lol
by mmoustafa - What are your plans to improve a substantial lack of customer support? https://news.ycombinator.com/item?id=49577159by frenchtoast8
- toven from openrouter, leading the team working with our providers - very interested in some of the things found in the report, I dug in to the image failures specifically, and in that case we have data showing deepinfra was correctly parsing images when the endpoint went live in july, but today fails those tests. we'll work on testing images and reasoning effort etc running constantly as chris mentions in point 3.
(screenshot showing our internal testing of deepinfra image inputs: https://raw.githubusercontent.com/ping-Toven/images/main/ima... )
by pingtoven - > We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them.
I'm glad to hear that y'all are doing this, as I was unaware that this was something OpenRouter does. I was surprised and disappointed that there are so many problematic providers that it seems like community best practice [1] is to ban somewhere in the realm of 5-6 providers. Would it be possible to provide some way to express an even stronger preference for high quality providers? E.g. "only route to first party for this model" or, "cost, but don't route to providers that more than x% worse than the first party". I'm sure something like that can be done via the API but I haven't found a UI way to do it - and having it in the UI would go a long way towards feeling like OpenRouter is looking out for me/helping solve the problem as opposed to leaving it to me to have to figure out.
[1] https://www.reddit.com/r/LocalLLaMA/comments/1mk4kt0/be_care...
by mixu - This is very useful information and comes at a perfect time! I use Openrouter for my newly released running tracker (I use it for live coaching and post-run debriefs). I've benchmarked a bunch of models over time to evaluate their aptitude for this specific task, and have noticed that sometimes a model can underperform for seemingly no reason. I'll be sure to include model providers in my benchmarking suite going forward!by rolfus
- Whats the tracker called? I'd like to try it.by aembleton
- Yeah, this is pretty accurate. Some providers are basically scams too. I encountered one provider for GLM-5.2 which ran at 200 tps (absurdly high), and was so broken that it would issue 20 full reads of the same file in one turn and quickly jump up to 300~400k context and charge me for the prefill. Went straight on my deny-list, but they got a few dollars out of me first.
Another common annoyance is having a request go to a provider that dribbles out ~1 tps (even for small models like DeepSeek V4 Flash). If you cancel the request, you still get charged for the prefill and the handful of generated tokens. If you don't cancel the request, you might be waiting 10 minutes for the turn to finish.
The overall experience is pretty good, and it's the best way to try new models, but they don't appear to do any real vetting or apply any quality standards to their providers, and occasionally it bites you.
by wren6991 - The best part about OpenRouter is 200 OK is probably hardcoded into their responses.
I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
by neya - I was just thinking that this kind of behavior is a glimpse of the future!