要点

The note discusses “streaming experts” for Mixture-of-Experts LLMs, where expert weights are streamed from SSD per token to run models that otherwise wouldn’t fit in available RAM.
It highlights rapid progress: Qwen3.5-397B-A17B running in 48GB of RAM is followed by reports of running a 1T-parameter Kimi K2.5 with 32B active weights in 96GB RAM on an M2 Max MacBook Pro.
The author also points to demonstrations of running the same Qwen3.5-397B-A17B model on an iPhone, though at low throughput (~0.6 tokens/second).
The post argues the approach is promising (“has legs”) and expects continued optimization via ongoing “autoresearch loops” to improve performance further.

Simon Willison’s Weblog

Sponsored by: WorkOS — The infrastructure fast-growing B2B companies use to sell to Enterprise.

24th March 2026

I wrote about Dan Woods' experiments with streaming experts the other day, the trick where you run larger Mixture-of-Experts models on hardware that doesn't have enough RAM to fit the entire model by instead streaming the necessary expert weights from SSD for each token that you process.

Five days ago Dan was running Qwen3.5-397B-A17B in 48GB of RAM. Today @seikixtc reported running the colossal Kimi K2.5 - a 1 trillion parameter model with 32B active weights at any one time, in 96GB of RAM on an M2 Max MacBook Pro.

And @anemll showed that same Qwen3.5-397B-A17B model running on an iPhone, albeit at just 0.6 tokens/second - iOS repo here.

I think this technique has legs. Dan and his fellow tinkerers are continuing to run autoresearch loops in order to find yet more optimizations to squeeze more performance out of these models.

Posted 24th March 2026 at 5:09 am