Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is really cool - I really like the beam search idea,
- beam search works when you have enumerable branches or some predefined dimensionby kimjune01
- this is the first time ive heard of beam search. i would have reached for a genetic algorithm of some sort, although it seems like some stochastic versions of beam search exist to avoid local minima. i wonder if there are any good frameworks for building these that agents can construct and use.by oinoom
- Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.by Jackobrien
- Yeah, but here's a dirty little secret that very few people are discussing:
You can't use Claude for this sort of thing if the goal is to make better AI systems. Anthropic finetunes Claude to dissuade people and the agent from using research that actually works. Anything that they use internally in their own models is poisoned, to protect their moat.
By proxy, that also means any openweights model that was distilled from Claude is equally useless for this purpose.
Thankfully, I don't believe OpenAI does this - they are far more honest and seem to care about their reputation. Anthropic is evil though.
by nullbio - This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?by dzbarsky
- some of these submissions seem to be omitting the actual rules. the #1 on edinh has a line that says "bypass ban check"by ramon156
- submission #2 by gau nernst is most numerically stableby dejavucoder
- it is nice to see continued enthusiasm for kernel programming and optimization. however, it would be nice if they have more eyeballs at the rocm side of things. gpu mode does have some challenges for amd's platform, but it seems sparse in comparison.by rldjbpin
- Mirrors my experience: LLMs are really good at optimizing, better than most humans. But also, they tend to not reach absolute peak performance where people made an effort to optimize something.
Since most problems see fairly little optimization, that's still a big win most of the time.
by bla3 - I've had pretty good luck with the following process for performance optimization loops:
- Have an agent generate unit tests until it gets to 100% path (not just statement) coverage, with every numerical test asserting checks against golden values to prevent regressions
- Let it rip on a performance improvement loop, for the widest E2E representative test case you have. Have it generate flamegraphs along the way so you can check in and steer it as necessary.
- Optionally allow for 1 ULP changes in output values so that it doesn't kill itself getting bit-exact results.
- Have it flag correctness errors as it goes, since your code probably isn't bug free.
This is also how I've done language ports from python to rust, and having the ironclad test coverage protects you from drifting.
by themeiguoren - Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sureby amarcheschi
- For once Cholesky is less general than QR: Cholesky only makes sense for positive matrices, while QR works for any matrix (including non square).
Also QR is a primitive for operations like finding eigenvalues, and I don't think Cholesky can be used there.
by cdavid - Meta commentary but it felt fresh to read a long wall of text that didn't seem to be AI generated. Thanks.by sqquima
- author here!
welcome! check out my featured section
by dejavucoder - How is this meta commentary when this or its negative version is present on literally every post.by halJordan
- Training material seems to be especially rich re GPU kernels and SIMD.
I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?
by tosh - They are easily verifiable and hill-climbable.
Because pre-LLMs humans partially "autogenerated" kernels through hyperparameter search and in some sense eating the code complexity in return for performance, and thus built tools for the same automatic verifiability that is useful for LLMs.
In some other tasks, we never built the same level of automatic verifiability since the level of automation in creation being much lower meant it's not giving you as much of a marginal benefit. We prefer code readability and simplicity and such in say, web services, because, say, the database IO time is going to dominate. Here getting an LLM to write a cromulent C# web service is more difficult since it's not easy to automatically verify whether code is cromulent or not. So if you put up LLMs to it, you end up with slop (which works).
OTOH, in kernel design, you give it access to every perf counter, every observable possible and have it optimise all of them. And all are verifiable/hill-climbable and you generally don't give a crap if the code is readable or reusable.
- Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its training. However, the LLM was able to derive a full listing of the operations and their arguments from gcc to get us started, and that's why I was able to come up with my own implementation. Along the way, it also came up with insights about possible gotchas. Then, when implementing algorithms, it has been able to reason things out and get things working, despite the ISA not being extremely well known.by dsign
- Well, GPU kernels are co-designed really hard. A lot of it is, async tile pipelines + spam my MMA primtives.
Obviously it's still hard, but the point is that, by construction (cause like, NVIDIA literally releases primitives like this, and/or people like TK build slightly higher-level primitives over the base hardware primitives), if you learn the complicated language, you can get really good results, and on some level you "know" you're right by construction even before you go to the actual empirical tests (since you're operating over a higher-level "language", and not arbitrary byte accesses).
Honestly a lot of interfaces and frameworks you could argue are like that, so it's not really a point for GPU kernels relative to other things. But maybe a hint as to what I personally think is important in the AI era - finding the right cuts, the right high leverage abstractions, as otherwise AI is going to produce spaghetti nonsense.
by sigbottle - It's been fascinating doing a custom variant for GFQL, the first OSS embeddable Cypher property graph query engine for CPU+GPU -
- accelerated launch of our new backends like polars, including a new lazy mode & planner, which are fundamentally new paths
- while we initially aimed for top GPU benchmark scores, we now also maintain top CPU scores too!
Long-term, more interesting to me is this opens rethinking what it means to be a query engine. Right now we are making it the fastest in general, especially on workloads from our own use, major industry benchmarks, and our users. At the same time, similar to jit and multistage computing, we're looking at new ahead-of-time optimization techniques users can do that are more interesting than plugging in custom indexes. Essentially, if our agents can do fast specializations, there should be safe hooks that we can expose to our user's agents too!
by lmeyerov - One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.
The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.
The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.
by augment_me - Isn't generalization of solutions what JEPA is trying to solve?by mycall
- this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.
i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)
by dejavucoder - Test coverage is important. You need to test a variety of inputs.
Also "will always" is way overconfident. A year ago nothing close to this existed at all. Next year it will fill yet a different role.
by bonoboTP - This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.by musebox35
- There's only three kinds of optimization realizations
1) Realize that your problem isn't as general as it was
2) Realize that your problem has a better memory model
3) Realize that your problem can be parallelized further
Hyperoptimizing usually falsely leans on 1
by vatsachak - Overfitting to the input is part of the meta in this type of challenges.
The goal is not to create good, general or maintainable code. The only goal is to produce the fastest code.
by josu - In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.
Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot
by Almondsetat - It's very much how I've been using claude code in the past 5 month: brainstorm -> generate specs (constraints) -> generate exact plan -> implementation + review -> test / validate. The last stage is the most crucial one, and it's the most difficult to get it right for a complex solution (it's tough to cover every variation). But so far, it's been working great..by 3abiton