F1 · Cloud & Infra
Profiling llama.cpp with Arm Streamline
Overview
On an Arm Neoverse or Cortex-A platform, this guide builds KleidiAI-enabled llama.cpp, integrates Arm Streamline marker annotations and annotation channels, and captures an LLM inference timeline on the Arm CPU. The workflow splits token generation into Prefill and Decode, then observes the compute graph, tensor shapes, KleidiAI kernels and multi-threaded behavior, showing Prefill is GEMM-dominated while Decode is GEMV-dominated. This profiling method chains an open-source LLM runtime, Arm-optimized kernels and CPU-level performance data into an analysis baseline for locating bottlenecks with real execution evidence.
Scope of verification
- It is an llama.cpp profiling reference workflow on an Arm CPU, using the specified llama.cpp b6202, KleidiAI build, Streamline annotations and gator capture to validate that Prefill, Decode, operator and multi-threaded behavior can be observed.
- Dependencies include an Arm Neoverse or Cortex-A Linux/Android platform, Arm Performance Studio or Development Studio, a matching gator version, PMU access, and a GGUF model runnable by llama-cli.
- Scope centers on profiling instrumentation and execution-behavior analysis; observations vary with model, prompt, llama.cpp version, build flags, thread configuration and hardware, and do not cover automatic performance optimization, cross-platform benchmarking or production workload tuning.
Canonical source: https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama_cpp_streamline/
This page is a guided summary with boundary notes, not an official reproduction.
This page is a guided summary with boundary notes, not an official reproduction.