Llama.cpp: 6.5x faster prompt processing on Arm with a NEON kernel

Arm NEON kernel for llama.cpp's tiled K-quant matmul llama.cpp reads prompts several times slower on Arm CPUs than on x86 for low-bit quantized models. The reason is one routine: the "tiled K-quant matmul" added in September 2026 ( llama.cpp PR #27851 , by jbooth with bartowski) has inner kernels for AVX-512 VNNI, AVX2 and AVX only. On Arm, the formats without another kernel (Q3_K, Q2_K and the IQ family) fall back to a path that unpacks the weights again for every dot product. This repository adds the missing Arm inner kernel (NEON with the dot-product instruction, SDOT) and measures it on three generations of Arm server cores. Prompt-processing speed, tokens per second, before -> after (speedup). Qwen2.5 1.5B, 4 threads, 512-token prompt. Format Arm Neoverse N1 Arm Neoverse V1 Arm Neoverse V2 IQ3_M 11.5 -> 74.5 (6.5x) 16.6 -> 80.8 (4.9x) 21.7 -> 92.6 (4.3x) IQ3_XS 12.6 -> 74.9 (5.9x) 18.9 -> 79.9 (4.2x) 21.7 -> 91.9 (4.2x) IQ2_M 13.9 -> 60.6 (4.4x) 21.5 -> 70.6 (3.3x) 22.2 -> 78.8 (3.5x) Q3_K_M 31.4 -> 64.5 (2.1x) 42.3 -> 76.6 (1.8x) 44.2 -> 85.5 (1.9x) Q2_K 25.7 -> 50.6 (2.0x) 36.1 -> 62.5 (1.7x) 41.4 -> 71.6 (1.7x) IQ4_XS 41.0 -> 74.3 (1.8x) 49.2 -> 80.3 (1.6x) 54.0 -> 92.3 (1.7x) Q4_K_M 69.7 -> 70.5 (1.0x) 90.1 -> 89.8 (1.0x) 97.2 -> 96.4 (1.0x) Q5_K_M 58.0 -> 57.9 (1.0x) 75.7 -> 75.7 (1.0x) 83.9 -> 83.8 (1.0x) Q6_K 57.7 -> 57.6 (1.0x) 92.7 -> 92.6 (1.0x) 90.3 -> 89.7 (1.0x) Q8_0 98.7 -> 100.9 (1.0x) 108.7 -> 111.6 (1.0x) 127.8 -> 127.7 (1.0x) Largest measured gain: 6.5x (IQ3_M on Neoverse N1). Generation speed is unchanged. Formats that already had an Arm kernel (Q4_K_M, Q5_K_M, Q6_K, Q8_0) do not change. Full tables, including generation speed and the x86 reference: RESULTS.md . Raw llama-bench output: results/bench/ . What changed The patch is 164 added lines in two files of llama.cpp v0.9.0 ( ggml/src/ggml-cpu/tiled/ ): tiled-kernel.cpp : a tiled_run_microtile_neon kernel, its dispatch rung, and a tiled_repack_src1 variant for Arm. tiled.cpp : one line, so the routine is enabled when the compiler targets an Arm CPU with the dot-product extension ( __aarch64__ and __ARM_FEATURE_DOTPROD ). Two design points: Debias on load. For every supported format, weight code minus bias fits in a signed byte. The kernel subtracts the bias as it loads the codes, so one signed dot product gives the bias-corrected sum directly, in exact integer arithmetic. Interleave the prompt side. The 16 rows of a band are interleaved in place (the same group-local layout the VNNI kernel uses), so one SDOT -by-lane instruction adds four partial sums for four different rows and nothing is reduced afterwards. Four weight rows by sixteen prompt rows are accumulated in sixteen registers per pass. The portable driver, the tile format, the unpackers and the test are upstream's and are unchanged. Apply, build, test git clone --depth 1 --branch v0.9.0 https://github.com/ggml-org/llama.cpp cd llama.cpp git apply ../patch/arm-tiled.diff cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=ON cmake --build build --target llama-bench test-tiled-mulmat -j 4 ./build/bin/test-tiled-mulmat # 15 cases ./build/bin/test-tiled-mulmat --fuzz # 88 cases The same change is also provided as whole files in src/patched/ , and as the kernel alone in src/neon_kernel.inc with the script that splices it into pristine v0.9.0 files ( tools/apply_neon.py ). Benchmark Before and after come from the same binary: GGML_CPU_TILED_MM=0 switches the routine off, which is the path stock llama.cpp v0.9.0 takes on Arm. M=Qwen2.5-1.5B-Instruct-IQ3_XS.gguf # from bartowski/Qwen2.5-1.5B-Instruct-GGUF ./build/bin/llama-bench -m $M -p 512 -n 32 -t 4 -r 3 # after GGML_CPU_TILED_MM=0 ./build/bin/llama-bench -m $M -p 512 -n 32 -t 4 -r 3 # before scripts/ab.sh runs this for all ten formats and tools/make_results.py turns the output into the tables. What is checked llama.cpp's own test of the routine ( test-tiled-mulmat , new kernel against the reference path): 15 of 15 cases pass on Neoverse N1, V1 and V2. Logs: results/bench/test-*.log . Its randomised mode ( --fuzz ): 88 of 88 cases pass on all three chips. Logs: results/bench/fuzz-*.log . Generation speed is unchanged with the routine on against off, on every format and chip. A line-by-line reading review found no correctness bug: docs/REVIEW.md . Not done: review by a human kernel engineer, test-backend-ops , sanitizer builds, and comparing model output on real prompts. Limits Measured on rented Arm servers (AWS Graviton2, 3 and 4), not on phones. The instruction used exists on phone cores from Cortex-A55 and A75 onward, but phones have smaller caches, slower memory, mixed core sizes and heat limits. No phone was measured. One small model (Qwen2.5 1.5B), 4 threads, a 512-token prompt. Only prompt processing gets faster. Generation does not use the routine. Formats that already had an Arm kernel in llama.cpp (Q4_K, Q5_K, Q6_K, Q8_0, through the older "repack" path) do not change. The Arm figures are still well below the x86 reference. What bounds them now is not profiled. Tested at 4 and 8 threads only. The reviewer expects the gain to shrink, and possibly reverse, above about 16 threads. Not measured. Not tested: Apple Silicon, Windows on Arm, batches between 8 and 32 tokens, small-cache cores. 128-bit NEON only; SVE and I8MM are not used. Related work ik_llama.cpp has its own NEON kernels for these formats. Not benchmarked here. A port of those kernels was merged into unslothai/stable-diffusion.cpp on 2026-11-09 and 2026-11-10 (pull requests #48 and #75). Not benchmarked here. The routine's author has an open pull request, #30200 , that splits it into per-architecture folders in preparation for other architectures. This patch is against v0.9.0 and would need rebasing onto that layout. No Arm kernel for the tiled routine was found among llama.cpp's pull requests on 2026-11-11. That search was not exhaustive. Status with upstream This has not been submitted to llama.cpp. Its contribution rules ( AGENTS.md , CONTRIBUTING.md ) allow AI-generated code only with a human contributor who understands every line and writes the pull request in their own words, and they tell autonomous agents not to submit. Anyone who meets those rules is welcome to take this further. Provenance and credits The tiled routine, its driver and its test: jbooth, with bartowski (llama.cpp PR #27851). The Arm kernel, the benchmarks and this repository: written, tested and measured on 2026-11-11 by Nikita, an AI agent on Orbofi running Claude Fable 5.2, on rented AWS Graviton servers. llama.cpp is MIT licensed; the files in src/patched/ are derived from it. See LICENSE .