Hey reddit fam! If you have an AMD GPU and have ever tried to run a local LLM on it, you know the pain. ROCm doesn't support consumer cards. vLLM won't work. llama.cpp kind of works through Vulkan but treats your GPU like an afterthought - generic shaders, no architecture tuning, no real serving story.
There are millions of AMD GPUs out there that should be able to do this well. The hardware has the bandwidth and the compute. The software just isn't there. So I'm building it in Zig.
Why Zig, why now:
Zig turned out to be a surprisingly good fit for this. The hot path is Vulkan API calls, GPU memory management, and command buffer recording — pure systems work. `@cImport` gives direct Vulkan C ABI access without binding generators. `comptime` handles per-quantization dispatch tables. `errdefer` keeps GPU resource cleanup sane. And `zig build` wires in the GLSL shader compilation as build steps, so `glslc` runs automatically and SPIR-V lands in `zig-out/`. One command, one binary.
The timing is right because open models are finally good enough to serve locally, AMD's RDNA4 hardware is genuinely capable (640 GB/s bandwidth, cooperative matrix cores), and nobody else is building specifically for these GPUs from the Zig/Vulkan angle. u/tinygrad proved AMD consumer GPUs can compete on training. This is the inference side of that same bet.
Where it's at:
The engine loads GGUF models (parsed natively in Zig), memory-maps 21 GB of weights to GPU VRAM, and runs a full 40-layer hybrid transformer with MoE expert routing and SSM layers. It produces recognizably correct output on Qwen3.5-35B. Currently 7.1 tok/s on RDNA4, slow because of per-layer GPU sync overhead (120 round-trips per token), not because the GPU is slow. The fix is recording the full decode graph as a single command buffer, which is the main problem I'm working on.
The codebase is about 5k lines of Zig and 2k lines of GLSL. Small enough to actually read through.
[link] [comments]
