Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Good luck getting it to run with NVFP4 on a DGX Spark, the very architecture Nvidia created that format for.by halfdeadcat
- Can you elaborate on the problem? The article claims support for dgx sparkby Scaled
- big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folksby macwhisperer
- Nemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb?by jadbox
- Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
- The new Meta 30B models seems A LOT better:
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
by XCSme - lightning is sparse, glimmer is denseby khimaros
- At what cost difference?by Tactical45
- and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW
- The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.by eli
- Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old)
but
Qwen 3.8 27B is dropping this week...
by thehamkercat - An open source model from Nvidia is a free drug to later buy their chips.
- I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
- 24 comments so far about Nemotron on this earlier submission: https://news.ycombinator.com/item?id=49257947by WalterGR
- They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
- I would like to propose something:
- problem: massive deluge of information because of AI
- solution: human beings should adopt a minimalist style of communicating in writing.
- e.g. this entire website page can be ten bullet points.
- hell yeahby sp1nningaway
- it takes more effort to write with concisenessby ashu1461
- In my opinion: the only way forward is zero-knowledge-proof authenticated social media.
We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.
If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
by encrux - Judging by how my nieces and nephews text this has already happened!by jamiek88
- kby fooker
- Why use more words when few do trick?
- While I agree with the spirit, I don't think the solution to bad prose is slightly less bad prose. We can write good prose instead.by stavros
- There's a famous Pascal quote, "I have only made this letter longer because I have not had the time to make it shorter."
Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.
by AlexB138 - > NeMo Switchyard, an open source library for smart routing
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
by thehamkercat - Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
by quinncom - The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
- https://github.com/NVIDIA-NeMo/Switchyard#routing-strategies
Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.
The nice thing is that it makes sense for specific setups, less conversation oriented.
As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.
You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
by rufasterisco - I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).
But yeah I'm skeptical all this overhead is worth it.
by eli - One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gainsby jmward01