Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratchby jdw64
- the real lesson: GPUs win on memory bandwidth not just FLOPs. batching ops keeps VRAM fed at 2TB/s instead of tripping to RAM at 50GB/s for every operationby cold_harbor
- How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?by big-chungus4
- I feel like there is no portable advice for performance. A torch model exported as onnx is a different model.
That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep.
And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess
by ThouYS - I'd want to see more about the failure modes. Production systems need graceful degradation more than optimal performance.by xiaod
- > in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS
wild
by tosh - One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.by liuliu
- This post is a classic! Also recommended: Horace also gave a related talk (covering the high-level picture of modern ML Systems) at Jane Street in Dec 2024 https://www.youtube.com/watch?v=139UPjoq7Kwby ollin