Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.

    For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).

    I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].

    [1] https://arxiv.org/abs/2507.13575

    by avaer
  • Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide
    by danbrooks
  • Let’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.ai

    Which is within reach of some higher end consumer hardware, especially with layer offloading.

    You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?

    by Chance-Device
  • > Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight

    If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?

    https://news.ycombinator.com/item?id=49611128

    by adrian17
  • These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...

    Remember to clear the downloaded weights afterward.

    Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.

    by Aurornis
  • There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release!) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if you are GPU starved and want local, but you might have trouble finding things it is good at.
    by verytrivial
  • I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.

    I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.

    by miffy900
  • If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...

    This should work:

      cd /tmp

  # Get the Prism macOS runtime
  curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  ./llama-prism-b10685-7dffb15/llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768
    Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:

      uvx llm openai endpoint http://127.0.0.1:8331/v1 \
    --model bonsai-2-27b --responses hi
    That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".
    by simonw

Explore Birbla archives