Activation Exposure & Feature Interpretability for GGUF via llama-server

You can now capture per-layer activation vectors from llama-server during inference, train sparse autoencoders on them, discover which internal features correspond to specific behaviors (sycophancy, hedging, creativity, etc.), and extract those features as GGUF control vectors for real-time steering.

What this is:

A C++ patch to llama-server that adds `/activations` endpoints, plus a Python pipeline for the full SAE workflow. The patch is ~400 lines across 5 files and adds:

• `GET /activations`: query per-layer mean activations (with top-K filtering)
• `POST /activations`: enable/disable capture
• `POST /activations/collect`: stream full per-token vectors to a binary file for offline training

What you can do with it:

1. Monitor activations live: see which features fire strongest during a conversation
2. Collect training data: stream per-token activation vectors to disk while running inference
3. Train a sparse autoencoder: decompose activations into ~16K interpretable features (takes about 40 seconds on an RTX 3090)
4. Discover behavioral features: define phrase clusters ("sycophantic phrases", "hedging phrases", etc.) and find which features are unique to each behavior
5. Extract control vectors: turn discovered features into GGUF files you can load with `--control-vector-scaled`
6. Steer in real time: suppress sycophancy, amplify creativity, whatever you want, at the feature level

How it works technically:

The patch hooks into llama.cpp's existing `cb_eval` callback to intercept `l_out` tensors (layer outputs) during the forward pass. GPU→CPU copy via `ggml_backend_tensor_get()`, stored in a mutex-protected global struct. The binary collection format is dead simple: 16-byte header + float32 arrays, directly readable with numpy.

The SAE pipeline is standard: collect activations → train sparse autoencoder → probe features with behavioral phrase clusters → extract feature directions as control vectors. The interesting part is the inter-cluster differential scoring: instead of just finding "features that fire on sycophantic text," it finds features that fire *significantly more* on sycophantic text than on any other cluster, so you get specific behavioral features rather than generic language features.

PR + repo:

• llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/20785
• Companion repo with the full SAE pipeline, guide, and example clusters: https://github.com/hrhdegenetrix/llama-sae-feature-interpretability

The companion repo has a quickstart script, example behavioral cluster definitions, and a comprehensive guide covering the full workflow.

Notes:

• MoE models are *extremely* sensitive to control vector scales. Dense models (Qwen3-8B, 4096 embd) handle scales of 0.15-0.6 fine. Qwen3.5-35B-A3B MoE (2048 embd) needs 0.01-0.05 or output goes garbled.
• The eval callback registration had a bug where it only got set inside the graph-reuse branch: so capture silently stopped working after the first inference. Took a while to track that one down.
• You need ~500K tokens of activation data for a good SAE. Harry's DPO conversations are ~14K tokens each, so 20 rows gets you there.
• Persona DPO overfits by step 200 with small datasets. Step 200 was the sweet spot (~97% eval accuracy).
• SAEs are not the be-all, end-all of this process and in fact are one of only several pathways to feature interpretability, but they are a simple approach and the process should be fairly adaptable.

Enjoy!