Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Is there a non-slop version of this information available?

    I am reading up on GPU / ML micro architecture and am looking for some good sources.

    by throwa356262
  • I skimmed through it, what makes you think it is slop?
    by latenightcoding
  • There was this article recently, which I personally found interesting:

    https://news.ycombinator.com/item?id=47208573 Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering (maderix.substack.com) 376 points | 3 months ago | 122 comments

    by jval43
  • This scans very much as AI-written.
    by carbocation
  • why?
    by dkdcdev
  • This is obvious Claude slop writing, the author would be advised to use vale [1] with samples of their own writing as a guide.

    > Performance begins with the roofline. On the M1 the engine holds about 12 fp16 TFLOP/s of compute against a DRAM-bandwidth ceiling. The roofline has a ridge point near 141 FLOP per byte, a 2 MB working-set threshold, a 0.23 ms floor under any single dispatch, and efficiency near 0.37 picojoules per FLOP at the compute optimum. On a 256-channel 3x3 convolution it runs about 3.8 times faster than the same chip’s GPU and 9 times more energy-efficient. The roofline pairs the engine’s throughput ceilings with its measured power.

    > Reaching the engine is not the same as running an arbitrary graph on it. The operations the engine executes are distinct from the ones a capability bit only advertises. A feature attested in the hardware tables or accepted by the compiler frontend counts only once a compile-and-run confirms it, and several advertised operations, three-dimensional convolution among them, never lower to the engine at all. Weight compression on the direct path cuts bandwidth, not only stored size. On the unentitled engine, int4 lookup-table weights run about 2.37 times faster than fp16, and structured sparsity 1.55 to 1.64 times faster at 0.43 times the bytes.

    https://vale.sh/

    by thx67
  • This Neural Engine seems useless for LLMs. Trapped in the wrong architecture
    by brcmthrowaway
  • Apple is releasing CoreAI which is supposed to be optimized for LLMs and the transformer architecture.
    by satvikpendem
  • At the release of Apple Silicon, there was this repos https://github.com/hollance/neural-engine That reference lot's of discovery and reverse engineer on the ANE.
    by Narew
  • It does not seem to cover the Neural Accelerators, Apple's equivalent of the Tensor Cores. They only got released on M5 platform. This is probably the most important part to cover.
    by g42gregory
  • Neural accelerators are easy to use from Metal. They kick in automatically if you do a matmul using Metal Performance Primitives and you use bf16 or smaller (they don't seem to work in fp32).
    by sakras
  • Those are part of the GPU not the Neural Engine.
    by wmf
  • I've managed to successfully use the ANE to accelerate text-to-speech models on iOS (as an aside - this was much more straightforward than the equivalent on Android).

    I did however struggle to run a diffusion model on the ANE - but found that mlx-swift and iPhone GPU sufficed: https://www.duration.ai/blog/generating-images-with-a-2020-i...

    by sudb
  • If anyone is interested in doing something seriously useful with these neural cores, there is this incredible write up on getting ModernBERT running on them: https://stephenpanaro.com/blog/modernbert-on-apple-neural-en...

    Really wish this author would blog more, this piece is incredible and includes the code.

    Also ModernBERT is amazing if you haven’t used it before, worth spending time with - have used it myself for classification tasks and it’s very impressive.

    by kamranjon
  • This is so, so good!
    by nl