LLMs don't understand. It's mind-boggling to me that large parts of the tech industry think that.

Don't ascribe to them what they don't have. They are fantastic at faking understanding. Don't get me wrong, for many tasks, that's good enough. But there is a fundamental limit to what all this can do. Don't get fooled into believing there isn't.

It’s incredibly easy to get LLMs to do a lot of stuff that seems convincing.

They are literally trained for plausibility.

This is epic work. Would love to see more of it but I guess you're gonna take it the startup route since you have connections. Best of luck.

For instance, you might have an SEO expert on the team, but that alone won't guarantee top search engine rankings. There are countless SEO professionals and tools (human or AI-powered), and even having the best one doesn't eliminate the underlying challenge: business competition. LLMs, like any other tool, don’t solve that fundamental problem.

We don't even know the optimal algorithms! AlphaEvolve recently found "an algorithm to multiply 4x4 complex-valued matrices using 48 scalar multiplications, improving upon Strassen’s 1969 algorithm that was previously known as the best in this setting." - https://www.nature.com/articles/s41586-022-05172-4

You severely underestimate the landscape of possible implementations for these kernels. There are many ways of performing a matrix multiplication and predicting which one will perform best without running them all is nontrivial, even with perfect knowledge of the underlying system.

This is just a completely incorrect take, speaking as a former insider.

For register allocation and instruction selection, there is hope because it is FPT and there are algorithms to do it optimally in polynomial time, albeit with a large constant factor (FPT), making it impractical to apply to compilers as of today. For instruction scheduling, it is just too hard. If you read literature on scheduling algorithms, it is NP-hard for apparently simple instances, e.g., 2 parallel identical machines with no preemption and bounding completion time (https://www2.informatik.uni-osnabrueck.de/knust/class/), while actual microarchitecture is much more complicated than this...

Needless to say, these are already the simpler problems. The longer the program or the more profiling data you can optimize for, the more tricks you can throw at it, and most of them are NP-hard to optimize optimally.

Being NP-hard doesn't imply that you can't obtain the optimal result, but compilers that I know of do not implement them, because most users are not willing to wait for days for such a compilation to complete. Ideally, one should make something that can run on clusters of CPUs or GPUs to optimize this, and people having those clusters will typically be willing to do this because they want to optimize the program they later run on the clusters. However, to my knowledge, no one is working on this at the moment.

The implication was that the FP32 versions of these kernels have lagged behind the more popular versions. There was opportunity to translate the advancements from other kernels into these. Someone would need to look closely to see exactly what was done, but it’s premature to suggest anything like “new algos” or “no pre-existing solutions”

This is a great use case for LLMs, though. I often do something similar where I make improvements to something I use most frequently and ask an LLM to translate that pattern to other similar parts of the code.

That being said, it is cool if AI is enabling lower cost adoption of better more optimized kernels with less effort.

Read the damn comment you're responding to. There have been human written kernels for both fp16 and fp32 for a long time.

Here is the corrected version of your comment:

"Wow, so, you're basically saying the AI created the same but faster algos in a well known domain with established pre-existing solutions, whose overall impact on the runtime of practical workloads is insignificant? Awesome!"

I think this error is large enough that referring to it as FP32 is misleading.

Also, the performance gains do not translate to my RTX 3060M GPU (3.8 GFLOPS vs PyTorch's 5.3), presumably because it lacks the optimized hardware for half precision.

But on the plus side, the single file was very easy to adapt and the code is quite readable. I have seen much uglier kernels.

It's just like image generation: the first iteration is the worst it will ever be.

https://cvw.cac.cornell.edu/gpu-architecture/gpu-characteris...

A function that is meant to be executed in parallel on an attached GPU is called a kernel. In CUDA, a kernel is usually identified by the presence of the __global__ specifier in front of an otherwise normal-looking C++ function declaration.

There's little doubt in my mind that the authors of the blog post went after the largest performance delta between their best AI-generated code and the default pytorch implementation, without looking into real-world nuances like "picking a good kernel out of a zillion options" or "production-quality output accuracy". It's cool that this sort of thing is being researched, but the results need to be taken with a grain of salt. Maybe I'm too jaded.

IIRC there was another paper recently, with similar methodology about computing xAx. These papers produce algorithms which aren't empirically correct, but provably correct. They do this by operating on a graph data structure, which describes the algorithm and then verifying the algebraic equality to the correct result.

There is a substantial difference here. And I think utilizing algorithms which only are empirically correct can be dangerous.