GPU Kernels on an H200: Five Inference Problems, Faster Than SGLang

AI products · Creator, sole engineer · in progress

My entry in a public GPU-kernel competition: faster kernels for five parts of one model's inference - attention decode, a whole decode layer, prefill, the dense projections and mixture-of-experts layers with 4-bit weights - each timed on one H200 against SGLang's kernel in the same run.

  • 1.77× faster attention decode than SGLang
  • 316 / 822 competition rank

How it works

No H200 at hand: check runs double as timing probes, so nothing reaches the judge without passing locally.

  • 5 problems → Lead agent
  • Lead agent → Kernel file
  • Lead agent → Reviewers
  • Reviewers → Kernel file
  • Kernel file → Local check
  • Kernel file → SASS check
  • Local check → H200 judge
  • SASS check → H200 judge
  • H200 judge → 1.25–1.77×

The detail

All five beat the baseline: attention decode 1.77x, prefill 1.37x, mixture of experts 1.27x, projections 1.26x and the decode layer 1.25x. Every problem was scored within about 20 hours of the first submission, moving the overall rank from 506 of 667 to 316 of 822.

I had no H200, only a laptop GPU of the previous generation, and a daily quota of judge runs. So each submission is one Python file with CUDA compiled at load time and a Triton fallback, check runs double as timing probes, and nothing goes to the judge without passing locally. A lead agent worked overnight on a schedule, with one agent per problem and separate reviewers.

Two lessons from the hardware: FP8 mma.sync is emulated on Hopper, so native FP8 maths means wgmma; and the judge's older compiler allocated registers differently, so every kernel is now checked in SASS with three compiler versions.

CUDA C++, Hopper wgmma and TMA, CUDA graphs, Triton, PyTorch, FP8 and NVFP4.

What I did

  • Wrote CUDA and Triton kernels for five inference problems, all faster than SGLang on the judge (1.25x to 1.77x)
  • Built a fallback path so a compile failure on the judge never wastes a scored run
  • Used check runs as benchmarks: variant sweeps timed on the H200 and returned in the log
  • Ran a lead agent overnight with one agent per problem, separate reviewers and a judge-quota budget
  • Found that the judge's compiler clobbered a wgmma register and added SASS checks with three compiler versions

Stack

CUDA, Triton, C++, Python, PyTorch, GPU Programming, Performance Optimization, AI Agent Development

Case study: https://www.siddhvasudev.com/projects/gpu-kernels