llmspeed: Feel How Fast a Local LLM Runs Before You Buy the Hardware

A tool surfaced on Hacker News on October 17. It is called llmspeed, and its one-line pitch is precise: feel how fast a local LLM runs before you buy the hardware.

The method is fun. Pick a GPU or Mac and a model, and text streams onto the screen at the predicted speed — including the wait before the first token. Reading a number in a benchmark table and watching an answer type itself at that speed are different experiences.

The catalog is broad: RTX 3090, 4090 and 5090, RTX PRO 6000, Mac Studio and MacBook lines, DGX Spark, Strix Halo, AMD and Intel GPUs, and CPU-only builds — 30 hardware options and 15 popular models, across CUDA, Metal, ROCm, Vulkan and CPU backends.

What it shows

The feature set:

llmspeed (llmspeed.dev, MIT open source)

- Real-time simulation: after a first-token wait matching your prompt
  length, text streams at the predicted tok/s
- Race mode          : two setups side by side (e.g. 5090 vs Mac Studio)
- Memory fit         : per quantization, shows
                       [fits] / [partial offload] / [won't run]
- Shareable links    : any setup or race as a URL
- Scale              : 30 GPUs, Macs and CPUs by 15 popular models
- Stack              : Vite + Preact + TypeScript, static site (GitHub Pages)

Some examples give a sense of scale. All figures are llmspeed's own.

The tool's value shows up right here. The RTX 5090, the fastest consumer GPU, is overwhelming on 35B-class MoE models, yet cannot hold a 748B-scale model at all. Meanwhile the Mac Studio M5 Ultra 512GB runs that large model at 78 tok/s. The screen makes one fact obvious: the question is not a ranking, it is what you intend to run.

Where the numbers come from, and how much to trust them

To use the tool well, know the provenance. llmspeed labels every estimate.

[Trust labels]
Measured  : benchmarked on this exact hardware, model and quantization
Derived   : interpolated from measured benchmarks of similar setups
Projected : calculated from specs, e.g. for announced but unreleased hardware

Sources: r/LocalLLaMA, localscore.ai, llama.cpp discussions —
         community measurements, collected and verified

When no measurement exists, it falls back to first principles, and that part is educational.

[The fallback model]
- Generation speed is bound by memory bandwidth:
  every token reads the active weights once.
  For MoE models, only the active parameters count.
- Prompt processing (prefill) is bound by compute (FP16 TFLOPS).
- Both use per-backend efficiency coefficients calibrated against
  measured benchmarks, plus a slowdown as context grows.
- Partial offload is a hybrid model:
  the GPU-resident share streams at GPU bandwidth, the spilled
  share at an effective system-RAM bandwidth.
- Memory fit counts weights, KV cache and runtime overhead against
  usable memory: about 75% of unified memory on Macs and
  93% of VRAM on discrete GPUs.

Two concepts matter here. First, generation speed is bottlenecked by memory bandwidth, not compute. How fast weights are read from memory sets tokens per second. That is why an RTX 5090 (about 1,790 GB/s) manages only 46 tok/s on a dense 27B model, while MoE models with fewer active parameters run far faster. Second, an MoE reads the active parameters, not the whole model, so parameter count is not speed. The background is in Understanding MoE Active Parameters.

The limits are stated plainly too.

[Limits llmspeed lists]
- Single device only. Multi-GPU setups (tensor or layer split)
  are not simulated yet.
- Estimates, not guarantees. Drivers, engine versions,
  flash attention, power limits and thermals can move real numbers
  by 10-30%.
- Projected values for unreleased hardware will be replaced with
  measurements as they appear.

Read hardware on two axes

What llmspeed ultimately exposes is two axes.

[Axis 1] Capacity - does this model fit in memory?
                   (fit / offload / won't run)
[Axis 2] Speed    - if it fits, how fast?
                   (bandwidth decides)

The axes move independently.
- RTX 5090 : top speed, capacity up to 32GB
- Mac Studio M5 Ultra : huge capacity, speed only as fast as bandwidth
- DGX Spark : holds large models, slow on dense ones
              (e.g. Llama 3.1 70B at about 4.4 tok/s)

A common buying mistake is picking a card on speed alone. If the model you want does not fit in that memory, speed means nothing. Conversely, even if it fits, whether the speed is usable in your workload is a separate question. For example, 12 tok/s is fine for a human reading along, but a bottleneck for an agent loop that calls the model many times per second.

Conclusion: a checklist before you buy

The briefing's action, spelled out:

[Checklist before buying local inference hardware]

1. Compute the memory requirement of your candidate model and quant
   [ ] total parameters + KV cache + runtime overhead
   [ ] for MoE, check active parameters too (they affect speed)

2. Look at capacity and speed separately
   [ ] the VRAM (discrete GPU) vs unified memory (Mac) trade-off
   [ ] time to first token (TTFT) vs generation throughput (tok/s)
       - with long prompts, TTFT drives how it feels

3. Check the labels in llmspeed
   [ ] is this cell Measured, or Derived/Projected?
   [ ] if only Projected, hold the decision and wait for measurements

4. Use race mode on two candidates
   [ ] same model, same quant — which one matches how you work?

5. Confirm with a real measurement
   [ ] llama-bench -m model.gguf -p 512 -n 128
   [ ] submit the result as a PR and the next estimate gets better

One line captures it.

Local LLM performance is decided first by
"does it fit in memory and does bandwidth keep up,"
not by how smart the model is.

llmspeed is interesting because it teaches that principle as an experience rather than a table. Knowing it before you buy — and after — cuts the cost of trial and error sharply. For the surrounding economics, see The Local AI Economics of DGX Spark 64GB and A Guide to AI Model Speed and Latency.

Further reading

• The Local AI Economics of DGX Spark 64GB
• AI Model Speed and Latency: What Actually Matters
• What Are MoE Active Parameters?

References

• llmspeed official site
• llmspeed GitHub repository (MIT)
• r/LocalLLaMA (source of community measurements)
• llama.cpp (includes the llama-bench tool)

Go deeper with a course

Once you can run a local model, the next question is what to feed it and what to ask of it. Retrieval (RAG) design becomes the next bottleneck, and this is where the hardware instinct turns into production work.

• View the practical RAG design course ↗