Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
    by marketingan
  • So you created a new account to blatantly plagiarize another comment from this same page? What's even going on here?
    by fc417fc802
  • I wish more people pursued that approach to teaching neural networks.

    First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is.

    Then, teach the intuitions behind derivatives and gradient descent. You don't need the entirety of calculus, there's no benefit to knowing how a sequence or limit works if you ) only want to understand neural networks. With autograd, you won't be manually doing derivatives of weird functions either, so intuitive understanding is a lot more important than doing dozens of traditional calculus exercises on paper like it's the 1800s. You could probably explain the little bit of calculus you need in an hour or two, even to somebody with a 12-year-old's understanding of math and a good bit of programming knowledge.

    Only when people understand the training and inference, implemented with loops and descriptive variable names, teach the tensor, explain how a modern CPU and GPU works (because many programmers still think a modern computer is just a much faster 6502), and then teach the tricks we use to make it fast.

    by miki123211
  • Needs 2022 in title
    by axpy906
  • Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
    by jdw64
  • It’s just linear algebra. Work your way from feed forward to CNN to RNN to LSTM to attention then maybe a small inference engine. Kaparthy’s llama2.c is only ~300 lines on the latter and it pragma simds so you don’t need fancy GPUs
    by glouwbug
  • If you want a written resource I have a blog post about the mathematics behind building a feed forward from scratch, https://max-amb.github.io/blog/the_maths_behind_the_mlp/. Kinda focuses on translation from individual components to matrix operations.
    by max-amb
  • If you aren't already aware, Karpathy has several videos that could get you there in a few hours https://www.youtube.com/@AndrejKarpathy
    by kflansburg
  • the real lesson: GPUs win on memory bandwidth not just FLOPs. batching ops keeps VRAM fed at 2TB/s instead of tripping to RAM at 50GB/s for every operation
    by cold_harbor
  • How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?
    by big-chungus4
  • Yeah, that part should not be read literally; `x.cos().cos()` and `x1 = x.cos(); x2 = x1.cos()` both launch the same number of kernels (two in unfused/eager mode, one in fused/torch.compile, see this test notebook [1]). I think the author chained the two cos calls to symbolize the idea of combining them (without exposing the intermediate result), but chaining the two cos calls doesn't literally trigger operator fusion.

    [1] https://colab.research.google.com/drive/13a4Y-ko6QLMPAhBz64c...

    by ollin
  • It’s really not a concept you can express in idiomatic Python very easily. This comes from the actual generated assembly involving copies from global GPU memory into registers (slow, bandwidth saturates quickly) and back in between the cosines. If you can avoid the intermediate roundtrip that cuts the cost approximately in half.
    by vrm
  • The author forgot to add "fused" here, like they did in other parts of the same section.

    Non-fused:

      foreach i
    y[i] = cos(x[i])
  foreach i
    z[i] = cos(y[i])
    Fused, no intermediate variable:

      foreach i
    t = cos(x[i])
    z[i] = cos(t)
    The temporary "t" doesn't leave the GPU. Sweeping the array twice makes you twice as dependent on memory bandwidth.
    by teo_zero
  • I feel like there is no portable advice for performance. A torch model exported as onnx is a different model.

    That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep.

    And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess

    by ThouYS
  • That's interesting as I was considering GGUF --> ONNX conversions (via Olive), but if this creates unknown distortions in the effectiveness and stability, it might be a dead-end idea.
    by mycall
  • I'd want to see more about the failure modes. Production systems need graceful degradation more than optimal performance.
    by xiaod
  • > in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS

    wild

    by tosh
  • Single core vs multi core accounts for much of this
    by xyzsparetimexyz
  • Which, lets be honest, is probably still being orchestrated by Python somewhere.

    Python is 9.75 million times faster than Python.

    by itishappy
  • re comments:

    yes of course this is apples to oranges but that's kind of the point

    it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU

    the interesting thing is why that is so

    CPU vs GPU (latency vs throughput), boxing vs dense representation, interpreter overhead, scalar execution, layers upon layers, …

    by tosh
  • This statement makes zero sense
    by p1esk
  • Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing.

    "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)

    by patmorgan23
  • One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.
    by liuliu
  • This post is a classic! Also recommended: Horace also gave a related talk (covering the high-level picture of modern ML Systems) at Jane Street in Dec 2024 https://www.youtube.com/watch?v=139UPjoq7Kw
    by ollin