AWS published MoE kernel optimizations on SageMaker AI on October 13. No new model, just kernel work, with numbers self-hosting teams feel directly:

Throughput: up to 10% higher (H200, FP8 MoE)
TTFT: up to 9% faster (H200, 8-way parallelism)
Released: open-sourced into vLLM and LMI containers
Caveat: varies by model and hardware setup

Three tweaks, one by one.

1. Matrix swap: lie down for WGMMA (2–4%)

Hopper's WGMMA instruction expects at least 64 rows in the M dimension. MoE decode often processes single tokens, so M=1 leaves most Tensor Core capacity idle.

Expected: 64 full rows at max Tensor Core density
Reality: M=1, mostly idle

AWS flips the multiplication so the large expert dimension (N=8192) sits in the leading operand, clearing the 64-row threshold easily and smoothing async prefetch. Result: 2–4% end-to-end output throughput on large MoE models, in vLLM 0.24.0 and later.

2. Backend routing: Triton over CUTLASS (10%)

The biggest single gain is a config change, not code. Routing FP8 quantized MoE work to the Triton backend instead of the default FLASHINFER_CUTLASS path raised end-to-end throughput 10%.

CUTLASS: scaling, matmul, aggregation as discrete steps, global-memory roundtrips
Triton: one fused hardware pass, read HBM once, work in registers
Shines at: low concurrency, thin decode batches

Not "Triton is always faster." Backends differ by hardware, quantization, token counts, and concurrency. Read AWS's number as a Hopper low-concurrency FP8 profiling result. Applied via routing policy in the LMI v28 container.

3. Kill JIT recompilation: stop rebuilding every time (TTFT 9%)

A fused rotary-embedding plus FP8-quant kernel recompiled on every new token count, since Triton specializes integer args by default. On multi-GPU setups, the rest wait while one rebuilds.

Fix: do_not_specialize on variable-length parameters
Effect: recompilations gone, 9% faster TTFT on H200 at 8-way TP
In: vLLM 0.22.0 and later (PR #42849)

A few lines of annotations for faster first tokens.

What your team should do

If you serve models yourself, in order:

1. Check vLLM version (0.22+ ? 0.24+ ?)
2. For FP8 MoE, compare backend routing (Triton vs CUTLASS on your traffic)
3. Measure throughput and TTFT together (p95 and distributions, not means)
4. On LMI containers, move to v28+ for the routing policy

Attach the lesson from our AIPerf guide: servers passing uniform synthetic loads still collapse on real traffic. The sweet spot is where p95 bends in a concurrency sweep.

It connects to our MoE active-parameter guide too: even with small per-token active paths, serving efficiency is decided by routing, memory bandwidth, and parallelism. The kernel race is fought on exactly that layer.

CodeBridge mini lab: start with version numbers today

[ ] Are vLLM / LMI versions new enough to include the fixes?
[ ] Which backend does MoE work route to?
[ ] Throughput and TTFT measured on your traffic (not vendor figures)?
[ ] Does the low-concurrency decode condition match your workload?

The cheapest experiment is the second: flip the backend and replay the same traffic. No code changes.

Conclusion: half of model performance is engine work

The frontier race is half model, half inference engine.

WGMMA alignment 2–4%, backend choice 10%, killing recompiles 9%. None changed the model. If you run MoE, check versions and routing this week. The 10% may be sitting there.

Further reading

References

Go deeper with a course

Practicing architecture choices with token cost, latency, and throughput in view connects directly to this article's measurements. Learn the structures linking search and generation.