AI
Cohere's megakernel targets idle time in low-batch inference
AlphaSignal examines Cohere's open-source serving engine, which keeps an entire decode step in one persistent CUDA kernel. Reported gains reach 1.58 times vLLM throughput in a narrow H100 configuration, with smaller end-to-end improvements on coding and reasoning workloads. The useful engineering idea is reducing launch and synchronization gaps, not a universal replacement for existing inference engines. The release is specific to North Mini Code, BF16 and batch sizes up to eight.
Editorial summary · Publisher article
What matters
- The reported batch-one result is 292 tokens per second on H100.
- Persistent task scheduling lets ready operations proceed without waiting for an entire previous kernel to finish.
- Prefill still uses ordinary PyTorch kernels and pauses decoding; hardware, model and sampling support remain limited.
What makes the design interesting
Each resident block reads task descriptors and uses counters to track dependencies. That permits earlier work on ready attention outputs and prefetching of immutable weights. The article also describes a meaningful scheduling tradeoff: a tuned order performed better than an attention-first schedule, while dynamic expert routing limited the usefulness of generic schedule searches. For an engineering team, the experiment is evidence that serving overhead deserves measurement alongside kernel arithmetic, with applicability tested against its own traffic.
Context & caveats
- Performance figures are reported benchmark results, not measurements reproduced by this newspaper.
- The 1.58-times headline and 1.25-1.41-times end-to-end figures describe different tests.
Why this made the edition
Interactive editorial review of the supplied source; selected for substantive reader value with limitations retained.
Source material