Been building this for a while and finally cleaned it up enough to share.
voice-agents-from-scratch is a numbered, chapter-by-chapter repo that walks the full real-time pipeline:
- Microphone capture
- Whisper for STT
- Local GGUF LLM (via llama.cpp)
- Kokoro for TTS
- Speaker output
Everything streams - you don't wait for the full LLM response before TTS starts speaking. That's the part that makes it feel like a real conversation instead of a chatbot with a voice skin.
Chapters:
- Intro
- Audio IO
- Speech to Text (STT)
- Text to Speech (TTS)
- Full voice loop
- Real time systems
- Tools
- Personality
- Projects
Each chapter is a runnable script + a short CODE.md walkthrough. There's also a small shared library so you can see how the pieces compose into a real system, not just isolated calls.
Why fully local matters here: you can actually see where latency lives. Warm-up, first-audio time, streaming chunk size - these aren't abstractions when you're running it on your own machine.
I plan a deployment chapter, thinking of using modal.com for it, wishes and suggestions are welcome.
Repo: https://github.com/pguso/voice-agents-from-scratch
I originally wanted to publish this repo using Node.js, but the ecosystem in Node.js is really not ready. There is a very good Kokoro-JS npm package, but when it comes to Whisper support or audio processing in general there are no good options.
Happy to answer questions about the architecture or tradeoffs I ran into.
[link] [comments]
