Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- For LLM inference in production we use vLLM (with our own model).by rurban
- There are a couple of options. One good way to find inference providers for open models is through hugging face (https://huggingface.co). you can select a model and see which inference providers serve it. you can even access it through hugging face.
If you just wanted to test a model or have super light work you can get some free access to alot of open source models through nvidia (https://build.nvidia.com).
There is also Runpod and a few more.
Is there something in particular you need? if its just cost/performance/speed, check hugging face for there inference providers, it compares them against these three.
Although it's closed, openai's luna model is a great balance between cost/performance/speed.
lastly if you wanted to help shape a new inference provider, I am working on a more pro/consumer focused platform. It has a nice iOS app, platform etc. Models are limited currently and it's in beta. if you want to take a look it's called Ando and can be found here - https://andoai.xyz. (Its very early in its build, it will break, any and all feedback is appreciated)