Tuomas Laitila

Winning the GPU MODE eigh competition

Batched symmetric eigendecomposition on a B200, from 67 ms to 6.8 ms in fifteen days, with a fleet of coding agents sharing one GPU.

7 September 2026

I won GPU MODE’s eigh competition. The task is batched real symmetric eigendecomposition on an NVIDIA B200: take a batch × n × n FP32 tensor, return eigenvectors and ascending eigenvalues in the torch.linalg.eigh convention, and be faster than everyone else on the geometric mean of 13 benchmark cases. The baseline, torch.linalg.eigh, runs the suite at a 67.2 ms geomean. My final submission runs it at 6.8 ms on the competition’s grader.

I did not write most of the code. Claude and Codex agents did, working in git worktrees against one shared GPU, with me steering. This post covers both halves: what the kernel ended up doing, and what it took to keep a fleet of agents productive on one B200 for two weeks. The worklog and the results ledger from the campaign are the source for every number here.

Best-so-far benchmark geomean, July 2026
Log scale. Hover for what landed. The first ten days were measured on my own box with clocks locked at 1800 MHz. From July 12 the box was gone and every measurement came from the competition's remote B200 with unlocked clocks, which reads about 7 percent faster on the same code. The step at the seam is the clock, not a win.

The problem

The input is symmetric up to FP32 roundoff. The checker never compares against a reference solver, because eigenvectors are not unique. Instead it checks three matrix identities in FP64, each relative to the matrix’s L1 norm and scaled by n·eps:

gate residual tolerance
eigen-equation A Q − Q diag(L) 200 n eps
reconstruction Q diag(L) Qᵀ − A 400 n eps
orthogonality Qᵀ Q − I 100 n eps

Plus ascending order and no NaNs. The gates are deliberately loose so that FP16, TF32, and FP8 are allowed internally. That one design decision shaped the whole solution. Almost every late win came from moving a boundary between two kernels from FP32 to FP16 and paying for it with one cheap repair step at the end.

The 13 ranked cases:

case batch n what makes it hard
dense 20 32 launch-bound, 20 tiny matrices
dense 40 176 under-fills 148 SMs
dense 40 352 under-fills 148 SMs
dense, mixed, rankdef, clustered, lapack even 640 512 five cases, one saturated shape
dense, mixed, nearrank, lapack geometric 60 1024 four cases, 60 matrices on 148 SMs
dense 8 2048 eight matrices, most of the machine idle

Nine of the thirteen cases are n512 at batch 640 or n1024 at batch 60, so those two shapes looked like most of the score. But the geomean is a product, so a 30 percent win on n32 is worth exactly as much as a 30 percent win on n2048. That arithmetic decided where the last week went.

What torch does, and why it is slow

An nsys profile of the baseline settled the first argument. On torch 2.12 with CUDA 13, torch.linalg.eigh at n512 is not batched Jacobi. It is cuSOLVER’s classic three-stage solver: Householder tridiagonalization (sytrd), a divide-and-conquer tridiagonal eigensolver, and a Householder back-transform. The tridiagonalization is BLAS-2. Each column needs a symmetric matrix-vector product with the trailing matrix, and the next column depends on it. That is n serial symv calls per matrix, and cuSOLVER’s version runs at about 15 percent of the memory roofline for that operation.

That chain was the wall for the whole competition. Every profile we took, on every day, said the same thing: the geomean is a tridiagonalization problem. The reflector chain cannot be parallelized across columns, tensor cores do not help a matrix-vector product, and the only levers are how fast one column goes and how much other work can overlap it.

The final pipeline

The shipped submission is a single 13k-line Python file with the CUDA sources inlined as strings, generated from csrc/kernels.cu, csrc/bindings.cpp, and a Python template. It has about 45 kernels and 51 custom ops. Every shape gets its own route. For the three big shapes the route is the same algorithm as cuSOLVER, rebuilt piece by piece:

  1. Prescale. One fused pass computes each matrix’s max entry, divides by it, and writes an FP16 shadow copy of A next to the FP32 one.
  2. One-stage blocked Householder tridiagonalization. A latrd kernel reduces a 32-column panel (16 columns at n2048) with the trailing matrix streamed into shared memory as FP16 tiles by 2D TMA through a ring of mbarrier-guarded buffers. The symv inner loop uses Blackwell’s fma.rn.f32.f16, which multiplies two FP16 values into an FP32 accumulator with an exact product in one instruction. At n512 one CTA owns one matrix, which fills the machine at batch 640. At n1024 and n2048 the panel is spread over a cluster of 2 or 8 CTAs, with the rows of the symv split across the cluster and a DSMEM st.async plus mbarrier handoff once per column. After each panel a fused mma.sync kernel applies the rank-2k update to the trailing matrix in 64×64 tiles and refreshes the FP16 shadow. The last 128 to 288 columns skip the blocked machinery and run an unblocked lane-per-row reduction from shared memory.
  3. Tridiagonal divide and conquer. Leaves of 32 rows (22 on the small shapes) are solved by parallel Sturm bisection plus inverse iteration, one warp per matrix, with a residual check and a QL re-solve for the columns that fail it. Each merge level is two kernels: a prep kernel that sorts, deflates, and compacts, replacing a chain of about 130 torch launches, and a merge kernel that solves the secular equation with an FP32 packed-f32x2 seed and an FP64 polish and builds the eigenvector update. Merges too small to fill the machine run on 2- to 8-CTA clusters. Children pass between levels as FP16. The final level writes its eigenvalues sorted and its eigenvectors in sorted column order, so there is no separate sort and gather.
  4. Back-transform. The Householder reflectors are aggregated into compact-WY blocks, 4 to 16 panels per block, and applied to the eigenvectors as FP16 tensor-core GEMMs with FP32 accumulation, carrying Z in FP16 between applies.
  5. One Newton-Schulz step. Q ← Z (1.5 I − 0.5 ZᵀZ), also in FP16 with FP32 accumulation. This is the repair that makes all the FP16 above legal. Eigenvectors rounded to FP16 fail only the orthogonality gate, and one Newton-Schulz pass restores a 14× margin on every input family we could generate. Eigenvalues never go through FP16.
  6. CUDA graph replay. Everything above is captured into a CUDA graph at import time, one per (n, batch) and per slot of a small ring, and replayed on each call after a fused copy-and-prescale of the input into the captured buffer. At n512 the batch is split into three chunks whose graphs are joined into one DAG so one chunk’s D&C overlaps another chunk’s reduction. At n2048 the DAG has seven nodes, and the left half of the D&C tree runs while the right half of the matrix is still being reduced. Graph replay was worth 16 percent at n512, 5 percent at n1024, 17 percent at n2048, and 26 percent at n176.

The small shapes are the same algorithm in a different body:

Per-case timings from the final merged run on the competition grader, unlocked clocks:

case torch (ms) final (ms) speedup
n32 dense 0.306 0.048 6.4×
n176 dense 6.9 0.72 9.6×
n352 dense 19.7 1.99 9.9×
n512 dense 223 18.1 12.3×
n512 mixed 209 18.5 11.3×
n512 rankdef 213 19.0 11.2×
n512 clustered 150 4.95 30.3×
n512 lapack even 270 18.9 14.3×
n1024 dense 123 15.5 7.9×
n1024 mixed 119 16.0 7.4×
n1024 nearrank 120 16.0 7.5×
n1024 lapack geometric 113 15.5 7.3×
n2048 dense 173 25.8 6.7×
geomean 67.2 6.78 9.9×

The torch column was measured on my box with locked clocks and the final column on the grader with unlocked clocks, so the ratio flatters the final by about the clock difference. The last build measured on my own box was 8.42 ms locked, an 8.0× speedup in identical hardware state, and 7.95 ms at the grader’s clock.

Timeline

Fifteen days, 1,474 commits in the task directory, 836 ledger entries: 207 wins, 345 kills, 201 recorded facts, 46 neutrals. The days below are the ones where the frontier moved.

July 2 and 3: escape the library

The first day produced the dev loop (CUDA sources spliced into a single-file submission by a script) and the first custom kernels: a shared-memory Jacobi for n32 at 2.4× over torch and a 3-CTA cluster Jacobi for n176 at 1.9×. Day two added a block-Jacobi cluster kernel for n352 at 2.6×. Jacobi was the obvious first move for a saturated batch, and for n512 it also went in at 1.1×.

The important result of day two was a negative one. Three agents built the alternatives for the big shapes in parallel: spectral divide and conquer via the matrix sign function, a two-stage reduction (dense to band, band to tridiagonal), and a one-stage blocked Householder reduction. One-stage won at n512, 85.5 ms against 190 ms for two-stage, and spectral D&C passed every gate but ran at 213 ms against torch’s 122 ms at n1024. A custom tridiagonal D&C merge kernel went from a 1083 ms prototype to 155 ms to 52 ms in one day. The geomean stood at 53.9 ms.

July 4 and 5: the composed pipeline

Day three shipped the composed route (custom latrd reduction, custom D&C, WY back-transform) for n512, 180 ms to 138 ms, and for n1024, 118 ms to 91 ms. Jacobi was dead for the big shapes from that day. A row-unrolled latrd with an FP16 shadow copy of A took the reduction from 54 ms to 24 ms at n512. Glue fusion, a fused D&C prep kernel, and FP64-native secular solves each took a few more milliseconds. Geomean 24.9 ms.

Day four found the FP16 back-transform plus one Newton-Schulz step, which improved orthogonality margins while cutting time on all three big shapes, and gate-aware z-deflation in D&C, which cut the merge kernel by 25 percent. A 2D-TMA tile symv in the reduction took n2048 from 80.6 ms to 58.1 ms and n1024 from 36.7 to 30.1. The same day the grader’s toolchain turned out to be CUDA 12.9 rather than our 13.0, and a cluster size that was optimal locally was 30 percent slower under the grader’s compiler. Geomean 17.6 ms at the end of the day.

July 6: the diet

The best single day. Nsight showed the reduction kernel was issue-bound: its duration tracked dynamic instruction count almost one to one. So the day’s theme was deleting instructions without changing the math.

17.6 to 12.2 ms in one day.

July 7: graphs on the grader, and a compiler regression

The grader kills any CUDA graph capture that begins inside a timed call. Probes showed capture at module import escapes the check and replay inside timed calls passes, so precapture at import became the standard, worth about 1 ms of official geomean. Splitting the n512 batch into two independent graph branches gave 7 percent on all five n512 cases. n32 got a direct solver that replaced Jacobi: 0.132 to 0.088 ms, a 2.7 percent geomean move from the smallest case.

The grader also moved to CUDA 13.3 that day. Our first 13.3 build regressed 26 percent, from 11.0 to 13.9 ms. The root cause was a stale #if __CUDACC_VER_MAJOR__ < 13 around a dispatch table. Fixed, 11.3. End of day 10.8 ms.

July 8 and 9: the small shapes and the tails

With the big shapes’ reduction near its wall, the geomean arithmetic pointed at the small cases. Verify-then-repair let n176 and n352 run an FP16 fast route with a per-matrix residual check and an FP32 re-solve for the failures: n176 1.81 to 1.51 ms, n352 3.41 to 3.05. Collapsing the last 128 columns of the reduction into a lane-per-row shared-memory tail took another 20 percent off both. A multisection eigenvalue solver in the n32 kernel: 0.085 to 0.067 ms. Together the smalls took the geomean from 10.8 to 9.5.

July 9 was the day with the most kills on the ledger, 55, and also nine shipped wins: an mma.sync fused trailing update, a 6-CTA-per-SM configuration for it, a three-chunk DAG at n512, pre-issued TMA for the next column, and a set of D&C prep fusions that were bit-identical and 0.5 to 0.7 percent each. 8.75 ms.

July 10 and 11: calibration and the last local wins

On July 10 I checked the live board. The leader was at 7.2 ms official. Our own official score was 8.16 and our local locked number 8.75, so locked-clock numbers were about 7.5 percent pessimistic. That killed a set of “we are near the floor” conclusions that had been calibrated against a stale 8.4 figure and reopened every shape. I also lowered the held-out accuracy bar from 2× to 1.2× margin. The last local wins were small and many: a zero-fill removal in the reduction, an FP16 D&C carrier, fused verify norms on the smalls. The last local measurement was 8.42 ms locked.

July 12 to 14: remote only

After July 11 there was no local GPU. Every measurement from then on was a submission to the competition’s remote B200 through popcorn-cli: a cold nvcc build plus the benchmark, seven to nine minutes, on a shared and sometimes overloaded service with unlocked clocks. The same code measured 7.81 ms there. The whole workflow changed, and that is its own section below.

The wins in this phase came from the clustered n512 case, which had been running the general route at 19.4 ms and had the largest slack of any case. A dedicated route had been developed on a side branch for days and written off as a wash at 18.5 ms, and my own notes from July 13 said a geomean under 7.5 was not reachable. A Codex session restructured it that day, 19.4 to 15.0 ms, and then took it through about two dozen shipped iterations on July 14: shared-row padding, a sparse sketch shrunk from 16 nonzeros per column to 2, a packed-upper larft worth 10 percent on its own, a half-carried Gram, live-row compact-WY updates. 4.95 ms at the end of the day. The same day shipped a class of direct-FP16 wins across all big shapes: when a consumer is going to round a boundary to FP16 anyway, have the producer write FP16 directly. Each was worth 0.3 to 2.7 percent per shape. Remote geomean 7.81 to 6.78 ms in three days, 72 ledger wins and 274 commits on July 14 alone.

July 15 and 16: production port

The last two days were a Codex-driven rewrite of the shipped kernels into a standalone package without the graph machinery, benchmark-case routes, or static output buffers: a solver you can import and call on any batch size and any stream. It measures 8.0 ms warmed on a B200, and the code went from a 13k-line generated file to a 3.1k-line module. Graph replay was tried in that package too and measured 0.35 percent slower than eager, so the graph-free version is the production default.

What did not work

The ledger has 345 kills. The decision-grade ones, each with the mechanism that killed it:

The pattern across the list: most of these were killed on paper first, then re-litigated when the premise changed, and a few came back. The n32 direct solver had been killed on July 6 with the argument that its Gram-Schmidt was unconditional. It was not, and the same solver shipped on July 7 at 33 percent faster than the Jacobi it replaced. The clustered route was written off twice before it became the single largest win of the campaign.

Running agents against one GPU

One human, one orchestrator session, and subagents in their own git worktrees off main. The subagent templates were pinned to Claude Opus, with a few Fable agents on the hardest missions, and Codex sessions joined on July 13 and did the clustered route and the production port. The notes record 114 distinct Claude agent ids and 28 Codex ones. Five agent roles:

The tooling that made this work was small and mostly boring:

Agent slots were scored by benchmark weight times current milliseconds times estimated slack from a named floor, until the geomean arithmetic on July 10 replaced that with “percent per case, weighted equally.” A mechanism that had just won on one shape was transferred to its siblings before any new direction was opened. Long-horizon ideas that were not ready to ship lived on bet/* branches with a resumable status file, so an agent could pick one up cold. The clustered route was one of those bets.

When the GPU went away

From July 12 the only B200 was the competition’s own, reachable by popcorn-cli. The constraints inverted. Compute and tokens were abundant. Measurements were serialized, took seven to nine minutes each including the cold build, failed intermittently, and returned nothing useful on a build error through the CLI. The notes cite 574 distinct submission ids over three days.

The workflow became a shared runner script that submits once, captures the submission id early, kills the CLI’s fragile live poll, and polls the results endpoint directly with a five-minute request timeout, because the endpoint took over two minutes to answer and a short timeout had made every retrieval look like a failure. Submission id capture is serialized across agents with a directory lock so ids from concurrent agents cannot cross. A usage log records who submitted what and when.

Three rules replaced the local A/B:

  1. De-risk offline first. Lock the algorithm in numpy, and where possible prove a kernel change with a host-side schedule proof before spending a submission. The n352 panel change on July 14 came with a script that exhaustively matched the old and new owner schedules for all five panels and 512 threads, and only then went to the remote.
  2. Measure as a candidate-parent-candidate bracket. Unlocked clocks drift, so a single candidate run is not evidence. Every keep in the notes from this phase quotes three runs and divides the affected cases’ ratios by the untouched cases’ ratio to cancel fleet drift.
  3. An empty result is inconclusive, not a kill. The service sometimes returned a terminal status with no timings at exactly the 600-second mark. Several ideas got a retry on that basis and a few of them shipped.

What I learned about orchestrating agents

Epilogue

The production port is the same numerical kernels without the competition scaffolding: no graph, no static outputs, any batch size, any stream, 3.1k lines, 8.0 ms warmed on a B200 for the same 13 cases. A graph-replay variant exists as a separate package for services that can prepare shapes at startup. One adversarial arrowhead family from the LAPACK test set still fails the FP32 fallback at n352 in the production build, and that is the open bug.