Overview
A Go inference engine for sparse mixture-of-experts models, including DeepSeek V4 Flash, ported from antirez's C implementation. It provides a command-line chat client and an OpenAI-compatible API server, with CPU, CUDA and Vulkan backends. Metal support is experimental.
The default build has no mandatory C/C++ runtime dependency. Model weights are memory-mapped rather than copied into a separate loading buffer.
How it works
The inference pipeline reads model weights from disk through memory mapping and runs attention and expert layers using the selected compute backend. CPU execution uses SIMD kernels; GPU backends handle supported operations on the device. The command-line client and HTTP server share this inference engine.
Features
CPU kernels
SIMD matrix operations for CPU inference, including work shared with go-pherence.
GPU backends
CUDA and Vulkan compute paths, with experimental Metal support.
Sparse MoE models
Inference support for mixture-of-experts models, including DeepSeek V4 Flash.
OpenAI-compatible API
A /v1/chat/completions server with streaming.
Go build
Build the CLI with go build ./cmd/ds4chat, without a mandatory C/C++ runtime.
Memory-mapped weights
Accesses model weights through memory mapping; inference still needs to read the required pages from storage.
Architecture
Posts
2026-07-11
My AI Model Tier List for mid-2026
2026-05-10
Notes for May 3-10
2026-05-09
The Local AI Moat