LLMs on NPU¶
End-to-end decoder-only LLM inference (prefill + autoregressive decode) mapped to the AMD NPU2 in bf16 via MLIR-AIR. Coverage and performance below are refreshed nightly. Per-model details and source are in programming_examples/llms/.
Nightly LLM Benchmark (NPU2)¶
End-to-end LLM inference performance on the AMD Ryzen AI 5 PRO 340 (Krackan Point, NPU2) benchmark runner — 2×32 GB DDR5-5600 SODIMM — refreshed nightly. TTFT is time to first token (prefill latency); Decode is steady-state generation throughput.
Verify: 🟢 pass 🔴 fail ⚪ skipped. — not measured.
Decode throughput vs context (tok/s)¶
Steady-state decode throughput at increasing KV-cache depth.
The models below reimplement AMD NPU LLM designs originally developed by the FastFlowLM team, using the higher-level abstractions of the MLIR-AIR dialect.
| Model | 1k | 2k | 4k | 8k | 16k | 32k | 64k | 128k | Verify |
|---|---|---|---|---|---|---|---|---|---|
| gemma3_4b_q4nx (HF) | 20.63 | 19.45 | 17.50 | 14.57 | 10.89 | 7.04 | 3.62 | 2.02 | 🟢 |
| gemma4_e2b_q4nx (HF) | 22.75 | 22.10 | 20.92 | 18.83 | 15.73 | 11.84 | 7.92 | 4.02 | 🟢 |
| lfm2_1_2b_q4nx (HF) | 67.75 | 62.73 | 54.67 | 43.36 | 30.76 | 19.44 | 11.20 | 6.06 | 🟢 |
| llama31_8b_q4nx (HF) | 13.20 | 12.60 | 11.51 | 9.81 | 7.61 | 5.25 | 3.05 | 1.73 | 🟢 |
| llama32_1b (HF) | 9.35 | 9.07 | 6.79 | 4.54 | 2.84 | 1.18 | 0.50 | 0.22 | 🟢 |
| llama32_1b_int4 (HF) | 14.00 | 11.61 | 8.22 | 4.48 | 2.64 | 1.13 | 0.47 | 0.18 | 🟢 |
| llama32_1b_q4nx (HF) | 68.77 | 63.88 | 55.40 | 43.83 | 31.00 | 19.53 | 11.24 | 6.07 | 🟢 |
| llama32_3b (HF) | 4.32 | 3.58 | 2.68 | 1.71 | 0.83 | 0.37 | — | — | 🟢 |
| llama32_3b_q4nx (HF) | 28.13 | 25.80 | 22.17 | 17.25 | 12.01 | 7.46 | 3.99 | 2.15 | 🟢 |
| phi4_mini_q4nx (HF) | 23.58 | 21.69 | 18.71 | 14.71 | 10.31 | 6.45 | 3.43 | 1.84 | 🟢 |
| qwen25_0_5b (HF) | 10.70 | 10.99 | 9.61 | 5.65 | 3.99 | 1.77 | 0.75 | 0.36 | 🟢 |
| qwen25_1_5b (HF) | 7.09 | 6.27 | 5.03 | 3.24 | 1.68 | 0.80 | 0.41 | 0.21 | 🟢 |
| qwen25_3b (HF) | 4.07 | 3.55 | 3.08 | 2.07 | 1.07 | 0.48 | 0.24 | 0.10 | 🟢 |
| qwen25_3b_q4 (HF) | 21.72 | 20.03 | 17.06 | 13.21 | 9.14 | 5.65 | 3.20 | 1.72 | 🟢 |
| qwen25_7b_q4nx (HF) | 13.39 | 12.85 | 11.87 | 10.30 | 8.16 | 5.76 | 3.63 | 2.09 | 🟢 |
| qwen3_0_6b (HF) | 8.59 | 6.44 | 4.41 | 2.88 | 1.29 | 0.55 | — | — | 🟢 |
| qwen3_1_7b (HF) | 5.78 | 4.92 | 3.34 | 2.52 | 1.18 | 0.54 | — | — | 🟢 |
| qwen3_4b (HF) | 3.09 | 2.47 | 1.85 | 1.31 | 0.58 | 0.24 | — | — | 🟢 |
| qwen3_4b_q4nx (HF) | 21.02 | 19.31 | 16.64 | 13.03 | 9.10 | 5.61 | 3.07 | 1.66 | 🟢 |
| qwen3_8b_q4nx (HF) | 12.71 | 12.09 | 10.95 | 9.26 | 7.07 | 4.75 | 2.79 | 1.56 | 🟢 |
| smollm2_1_7b (HF) | 5.70 | 5.00 | 3.52 | 1.96 | 1.29 | — | — | — | 🟢 |
— not swept at this context, or an expected failure.
Prefill latency vs padded prefill length (TTFT, ms)¶
Time to first token at increasing padded prefill length. Each column is a separate build of the prefill ELFs: the prompt is padded to the built length, so TTFT tracks that length rather than the number of real prompt tokens.
| Model | 512 | 1k | 2k | 4k | Verify |
|---|---|---|---|---|---|
| gemma3_4b_q4nx (HF) | 984 | 1751 | 3385 | 7096 | 🟢 |
| gemma4_e2b_q4nx (HF) | 828 | 1458 | 2481 | — | 🟢 |
| lfm2_1_2b_q4nx (HF) | 379 | 586 | 1020 | 81875 | 🟢 |
| llama31_8b_q4nx (HF) | 1264 | 2337 | 5420 | 11566 | 🟢 |
| llama32_1b (HF) | 280 | 496 | 929 | 1920 | 🟢 |
| llama32_1b_int4 (HF) | 297 | 496 | 936 | 1937 | 🟢 |
| llama32_1b_q4nx (HF) | 271 | 446 | 822 | 1786 | 🟢 |
| llama32_3b (HF) | 612 | 1067 | 2224 | 5065 | 🟢 |
| llama32_3b_q4nx (HF) | 635 | 1048 | 2219 | 5011 | 🟢 |
| phi4_mini_q4nx (HF) | 702 | 1213 | 2516 | 5751 | 🟢 |
| qwen25_0_5b (HF) | 357 | 486 | 807 | — | 🟢 |
| qwen25_1_5b (HF) | 703 | 1136 | 2115 | 4328 | 🟢 |
| qwen25_3b (HF) | 1147 | 2053 | 3980 | 8204 | 🟢 |
| qwen25_3b_q4 (HF) | 1125 | 2116 | 4044 | 8100 | 🟢 |
| qwen25_7b_q4nx (HF) | 2021 | 3835 | 7637 | 16925 | 🟢 |
| qwen3_0_6b (HF) | 379 | 636 | 1250 | 2672 | 🟢 |
| qwen3_1_7b (HF) | — | — | 1793 | 3773 | 🟢 |
| qwen3_4b (HF) | 1356 | 2393 | 4770 | 10183 | 🟢 |
| qwen3_4b_q4nx (HF) | 1346 | 2444 | 4641 | 9915 | 🟢 |
| qwen3_8b_q4nx (HF) | 1807 | 3304 | 6588 | 13514 | 🟢 |
| smollm2_1_7b (HF) | 462 | 816 | 1567 | 3276 | 🟢 |
— not swept at this length, or an expected failure.
Vision-Language-Action policies¶
Not autoregressive: these emit one action chunk per observation rather than a token stream, so there is no decode throughput and no context to sweep. Action chunk is the latency to the model's first usable output. Prefix is the token count the backbone attends over.
| Model | Prefix | Action chunk (ms) | Measured | Verify |
|---|---|---|---|---|
| smolvla | 241 | 642.6 | 2026-10-14 | 🟢 |
| smolvla_all | 241 | 392.8 | 2026-10-14 | 🟢 |
📈 Performance history over time — per-nightly TTFT and decode throughput plotted per model.
Last updated 2026-10-14 · mlir-air 0c9074e · mlir-aie 35fa6b1 · llvm-aie 22.0.0.2026100501+94b320c0