Fast code is designed, not rescued
There’s a scene I’ve watched replay too many times to still find it funny. An application ships, everyone’s happy, and six months later it crawls. So a meeting gets called. People say the word “optimization.” Someone profiles a bit. And then comes the discovery: to make it fast, you’d have to change the core data model — the one that two hundred files now depend on. Cost of the job: too high. Verdict: add more servers. Performance just lost. And it lost before anyone even brought it up.
I want to defend a simple, almost boring idea that we apply surprisingly rarely. Performance isn’t an optimization you bolt on at the end. It’s a way of laying out your data that you choose at the start. And the good news is that this discipline pays off everywhere — on today’s CPU and on tomorrow’s GPU alike.
The real problem isn’t compute. It’s memory access.
When a program is slow, intuition says “it’s doing too much math.” Most of the time, that’s wrong. It’s waiting. It’s waiting for data to travel from memory to the processor, and it’s waiting because you told it to fetch that data from all over the place, out of order.
Take the most classic example there is: an array of objects. In memory you’ve lined up your Particles, each with its position, velocity, mass, color, and three fields nobody remembers anymore. Now you want to update just the positions. But the processor doesn’t read “a position” — it reads a cache line, a contiguous 64-byte block. So for every particle, it hauls in the position and everything else it doesn’t care about, then jumps to the next one. You’re paying to move data you never use, on every single iteration.
Flip the structure around — an object that holds arrays, instead of an array that holds objects — and everything changes. The positions now sit side by side in memory, in the exact order you’re going to read them. The processor loads one cache line, and every byte it carries gets used. Same business logic, same result, and often a several-fold cut in runtime. You didn’t “optimize” anything. You just laid out your data the way you were going to use it.
What’s good for the CPU clears the path for the GPU
And here’s where it gets interesting. This discipline — flat data, contiguous, read in order — isn’t a CPU trick. It’s exactly what a GPU asks for too, for different but converging reasons.
Let’s be honest about the nuance, because an HPC architect will pull you up otherwise. The CPU and the GPU don’t love memory for the same reasons. The CPU wants a single thread to walk contiguous addresses, so it stays inside its cache line. The GPU launches thousands of threads at once, and it wants neighboring threads to touch neighboring addresses at the same instant — what we call a coalesced access. The detail of what you’re optimizing isn’t identical.
But the layout that makes both happy is the same: flat arrays, struct-of-arrays, no pointer chasing, no objects scattered across the heap. Code that hunts references all over the place is slow on the CPU — and on the GPU it’s simply impossible to accelerate. Code that thinks about memory layout from the start finds itself, almost by accident, ready for massive parallelism. You optimized for today, and you paid in advance for tomorrow.
Why “we’ll optimize later” almost never works
If designing for efficiency pays off so well, why do so few teams do it? Because the promise of “later” is comfortable, and because nobody likes slowing down at the start for a gain they won’t see right away.
The trouble is that “later” always shows up at the worst possible moment: when the data model is set in stone, when the business logic depends on it, when touching the structure means breaking everything. And that’s exactly when the rational decision becomes “no, we won’t do it.” After-the-fact optimization doesn’t fail because we don’t know how. It fails because by the time it’s needed, it has become too expensive to be allowed.
Designing for efficiency doesn’t mean micro-optimizing every line — quite the opposite. It means making, early, the handful of structural choices you won’t be able to undo easily later. The rest, the real fine-tuning, can wait for the profiler. But data layout gets decided at the beginning, or never.
Where we try to help
This same discipline shows up, untouched, the moment you actually move code to the GPU. What was merely a good habit on the CPU becomes an explicit constraint there: blittable structs, flat arrays, coalesced access. It’s exactly the list of traps you need to know to port C# to CUDA without banging into every wall.
Rather than keep that know-how in our heads, we distilled it into a Claude Code plugin, open source. It knows Hybridizer’s attributes, the restrictions on device-side code, the kernel patterns (reductions, shuffles, CUDA graphs), and the infamous gotchas list. It can analyze a C# function and propose the matching kernel shape, review a port against those traps, and drive the build end to end.
I call it a porting assistant, not an oracle, and that’s deliberate. It doesn’t replace the judgment of someone who knows their domain. It makes the move to the GPU less intimidating by packaging the experience instead of reserving it for insiders. It’s early — the content comes from a real port, TinyLlama inference at around 125 tokens/s on an RTX 5070 — and it keeps evolving. But the direction is right: you don’t help people lay out their data better by lecturing them, you help them by handing over the tools to make the jump when they decide to take it.
At the end of the day, it comes down to one line: you don’t run fast by starting to run at the finish line. Performance, like a clean codebase, is won the moment you lay the first brick. Everything else is catch-up — and we all know how catch-up ends.