Overview
This llama.cpp fork is a measured inference stack for two constrained machines: a LattePanda Sigma with an Intel Core i5-1340P, and SpaceMIT K3 boards with RVV and IME acceleration.
Optimisations are promoted only when complete model and service workloads improve. The repository records rejected kernels, offload paths and cache designs alongside the selected profiles.
How it works
On the Sigma, a native Clang CPU build runs mmap-backed target and assistant GGUFs on eight pinned P-core threads. The deployed Gemma service exposes an OpenAI-compatible loopback endpoint to Pi, with two independent KV streams, prompt reuse and bounded checkpoint storage.
The K3 backend adds RVV kernels, IME1/IME2 dispatch, TCM staging, AI-core affinity and load-time weight repacking. Platform-specific paths remain gated by output checks, complete-model throughput and service behaviour.
Features
Intel local provider
Gemma 4 E4B is the primary local model, with Qwen retained for repository-grounded work and Maple for fast prompt ingestion.
Long-context validation
Gemma and Ornith completed near-capacity 128K requests. The Qwen rollback profile accepted an uninterrupted 99,104-token input.
Expert I/O
Router-aware prefetching maps selected GGUF expert ranges, checks page residency and advises only nonresident data under bounded policy.
Speculative decoding
Target and assistant models use deterministic checkpoint restoration, model-shaped fixtures, semantic replay and per-phase telemetry.
SpaceMIT backend
Adds RVV, IME and TCM paths, quantised-weight repacking, routed-MoE handling and optional compact-IQ tile caching.
Measured rejection
Iris Xe Vulkan and SYCL remain validation targets; their local kernel wins did not survive end-to-end promotion gates.
Architecture
Posts
2026-07-17
My AI Model Tier List for mid-2026