Overview
This llama.cpp fork focuses on long-context local inference on two hardware platforms: a LattePanda Sigma with an Intel Core i5-1340P, and SpaceMIT K3 boards with RVV and IME acceleration.
The work includes CPU tuning, model loading and platform-specific kernels. Changes are compared using complete model workloads, since faster individual kernels do not always make inference faster.
How it works
On the Sigma, a native Clang CPU build runs memory-mapped GGUF models on pinned P-core threads. A local OpenAI-compatible endpoint serves Pi, with prompt reuse, separate KV streams and checkpoint storage.
The K3 backend adds RVV kernels, IME1/IME2 dispatch, TCM staging, AI-core affinity and load-time weight repacking. These paths are checked for output correctness and complete-model throughput.
Features
Intel local provider
Gemma 4 E4B is the primary local model, with Qwen retained for repository-grounded work and Maple for fast prompt ingestion.
Long-context validation
Tests cover long prompts and KV-cache reuse with Gemma, Ornith and Qwen.
Expert I/O
Router-aware prefetching maps selected GGUF expert ranges, checks page residency and advises only nonresident data under bounded policy.
Speculative decoding
An assistant model proposes tokens for the target model to verify, with checkpoint restoration for rejected proposals.
SpaceMIT backend
Adds RVV, IME and TCM paths, quantised-weight repacking, routed-MoE handling and optional compact-IQ tile caching.
GPU limits
Iris Xe Vulkan and SYCL remain experimental: faster individual kernels did not improve end-to-end inference on the Sigma.
Architecture
Posts
2026-07-11
My AI Model Tier List for mid-2026