 | Not a chatbot pretending. Not a lookup table with a trench coat. A proper decoder-only transformer. Attention, RMSNorm, feed-forward, residuals, the works. Two layers, four heads, about 25,000 parameters. All int8. Trained with quantization-aware training so the float model and the integer model agree on what the next token should be. It lives on a floppy. It takes more than a minute per token. A full reply is several minutes of waiting while the border flashes colors and the SID chip beeps once per token to tell you it’s still in there, still pondering! I’ve been sitting in the same room with it for days now. Occasional beep behind me. I still grin every single time it announces a token drop :D Well, admittedly.. it’s not exactly smart, but considering the fact that its 25,000 parameters are about 70 million times smaller than those of GPT-4 et al I think we can accept that. I trained my C64 on roughly a hundred short emotional-support exchanges (“i’m sad” -> “that sounds really hard”) and now it tries to be nice to me, in its broken little “me me, here here”-way. “HELLO! RE SOUNDS ME. MEFUL!” is arguably nonsense, but the intention somehow shines through.. Or its my mind tricking me into believing its deeper than it should? All I can say is that the first time I read it I felt a deep satisfaction and a childhood dream coming true..My C64 is alive now! Don’t ask me to defend that. I’m just reporting ;) 64k should be enough for every bot25 KB of weights on a machine with 64 KB of RAM. After you load them, there’s still room for the code, the activation buffers, the tokenizer tables, BASIC, the KERNAL, all of it. The C64 has actual slack left over after hosting a real transformer. In hardware from 1982. The trick is that every weight is a single byte. A per-tensor shift baked in during training lets int8 do the work that most frameworks hand to 32-bit floats. 4x less storage, 4x less bandwidth, and no accuracy cliff if you trained for it. The 6510 has no multiplier, no divider, no floating point. So every matmul is shift-and-add. Division is restoring long division. RMSNorm wants a square root, so there’s an integer isqrt. Softmax is a 128-entry precomputed exp table.. in pure assembly, all bit-exact against a Python reference before any of it touched my precious real hardware. Who needs NVIDIA anyway?The chip the C64 ships with can run the same architecture OpenAI or Google runs their models on. It’s just slower. Much, much much slower. Proudly slower. You can run your own AI chatbot on your own hardware! No excuses! :)This whole project started as a joke and turned into something I actually mean. Every headline about AI right now is about scale. Bigger models, bigger clusters, bigger data centers, bigger power draw, bigger water bills, bigger government contracts. Someone announces they’re buying the world supply of DRAM. Memory prices triple. They quietly walk it back. Prices don’t come down. Small builders everywhere get to clean up the mess. Retro repair folks can’t source chips. Game studios’ hardware budgets explode. The child who knocked the shelves over is already in the car. And then the same people turn around and tell you the future requires more muscle. More compute. More everything. Trust them, Bro! The singularity needs another hundred billion dollars and it also needs your grid capacity and also your groundwater. The future isn’t more muscle. The future is better thinking. A 25k-parameter transformer with a thoughtfully-trained tokenizer, sensible quantization, and honest arithmetic can have a (broken, tiny, sweet) conversation on a computer from 1982. Scale that insight up and you get models that are small enough to run on your phone, your fridge, your car, your Commodore, without anyone needing to own a power plant. The research is already pointing that way. Smaller models, better data, smarter training, sparsity, distillation. Every month there’s another paper saying “actually you can do this with a tenth of the parameters if you just…” We won’t get to find out where that road leads. Not really. Because the people with the money decided the answer was “more” before anyone finished the sentence. The billionaires eat all the cake. The rest of us get told the cake shortage is our fault and also here’s a subscription. Well, it doesn’t have to be that way.. and because actions speak louder than words: I put a real transformer on a 1 MHz Home Computer from the year E.T. came out, and I released it for you to experiment with it… Everything is on GitHub: https://github.com/gizmo64k/soulplayer-c64 .. weights, disk image... and soon the source, too [link] [comments] |
I put a transformer model on a stock Commodore 64
Reddit r/LocalLLaMA / 4/9/2026
💬 OpinionDeveloper Stack & InfrastructureIdeas & Deep AnalysisTools & Practical UsageModels & Research
Key Points
- A contributor reports running a true decoder-only transformer (two layers, four heads, ~25,000 parameters) on an unmodified Commodore 64 using int8 weights stored on a floppy disk.
- The model uses quantization-aware training so an integer (int8) version matches the floating-point model’s token predictions, enabling the transformer to fit in 64 KB RAM alongside system code and buffers.
- Inference is slow (over a minute per token; full responses take several minutes), with visible/color border effects and the SID chip beeping once per token while the model generates.
- The write-up emphasizes that the C64’s lack of floating point hardware requires implementing operations like matmul via shift-and-add, RMSNorm via integer square root, and softmax via precomputed tables in assembly.
- The project is presented as a “childhood dream” demo more than a practical chat system, contrasting its tiny parameter count with modern LLMs like GPT-4.
