Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/Nvidia

Join the discussion

• Hacker News
• Intel hardware can have Vulkan overhead, impacting performance.
  by DylanMerigaud
• This seems to be a thin wrapper around libllama, what's the point? Llama.cpp already ships a web server. I don't see anything in the README that llama.cpp doesn't already support.
  by kgeist
• > llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback

  I got excited about someone paying attention to intel. Oh well.

  by peddling-brink
• Curious about Vulkan overhead on Intel vs AMD/Nvidia for long context. Any benchmarks vs vllm/sglang?
  by antonyragleap
• I didn't see any benchmarks against vllm, sglang, exllama, etc
  by PcChip
• > Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback

  so this is just a go binary that exec's `llama-server --model $MODEL_PATH ...`? there's room for llama wrappers, sure, but the readme only shows features that are already exposed directly from llama-server. it doesn't seem to do anything besides rename the CLI arguments: i don't get it.

  by colinsane