For the past six months, I've been running an experiment: instead of asking my team to write documentation, I asked an AI agent to listen.
The result is a living knowledge base — not a static wiki that rots after two sprints, but a system that updates itself as we work. Here's how it works, what broke, and why I think this is the future of team memory.
The Problem: Documentation Dies in Silence
Every engineering team I've been on struggles with the same cycle:
- Someone writes a beautiful onboarding doc.
- Six months later, the architecture changes.
- The doc becomes a trap — accurate enough to feel trustworthy, wrong enough to waste a day.
By the time a new hire points out the inconsistency, the person who wrote the original doc has moved teams or left the company. The knowledge is gone.
The Shift: From "Write It Down" to "Capture the Stream"
Instead of treating documentation as a separate task, I set up an AI agent (built on a local LLM + MCP stack) to observe our natural work streams:
- Slack threads where we debate architecture decisions
- PR descriptions that explain the "why" behind changes
- Meeting transcripts from technical discussions
- Code comments and commit messages
The agent doesn't just archive these. It synthesizes them.
How the Pipeline Works
1. Ingestion
Every day, the agent pulls new messages, PRs, and transcripts. It uses semantic chunking to break content into topic-based fragments rather than storing whole conversations.
2. Classification
A lightweight classifier tags each fragment:
architecture-decisiononboarding-relevantapi-contractincident-postmortemdeprecated
3. Synthesis
Once a week, the agent generates "living pages" — structured docs that combine related fragments into coherent explanations. These pages link back to the original sources, so you can always verify context.
4. Stale Detection
The agent flags pages that haven't been reinforced by new sources in 30 days. If a page about our auth flow hasn't seen a related PR, Slack thread, or meeting in a month, it gets marked potentially stale — not deleted, but labeled.
5. Human Review Loop
Every Monday, the agent posts a summary in our team channel:
- 3 new pages generated
- 2 pages flagged as stale
- 1 conflicting explanation detected (two sources say different things about caching)
We spend 10 minutes reviewing. That's it.
What Actually Changed
New hire onboarding dropped from 3 days to 4 hours. Not because the docs are longer, but because they're current. When someone asks "why did we choose Postgres over Mongo?", the answer includes the original Slack debate, the PR that finalized it, and the meeting where we discussed the migration — all linked.
Decision velocity increased. We stopped re-debating questions we'd already answered. The agent surfaces past decisions when it detects similar keywords in new discussions.
Documentation guilt disappeared. Engineers don't feel bad about not writing docs because the system captures their natural explanations. The PR description they already wrote becomes the seed of a knowledge page.
What Broke (And How We Fixed It)
Noise overload. Early versions captured too much — every Slack joke, every "LGTM" comment. We added a relevance filter that requires at least one technical keyword + a minimum character threshold.
Conflicting truths. Two engineers would explain the same system differently in different channels. The agent now detects contradictions and surfaces them for human resolution rather than picking a winner.
Privacy concerns. Not everything should be captured. We maintain an explicit exclusion list (channels, DMs, certain keywords) and run the entire pipeline locally — no data leaves our infrastructure.
The "illusion of completeness" problem. A living knowledge base can feel too authoritative. We added clear markers showing the last time each page was reinforced by a primary source, so readers know when to dig deeper.
The Stack (If You Want to Build This)
- LLM: Local Llama 3.1 via Ollama (keeps everything on-prem)
- Vector store: ChromaDB for semantic search
- Connectors: MCP servers for Slack, GitHub, Zoom transcripts
- Scheduler: A simple cron job that triggers the pipeline
- Frontend: A static site generated from the living pages, hosted internally
Total setup time: one weekend. Maintenance: near-zero.
Why This Matters Beyond Documentation
This isn't just about docs. It's about organizational memory.
Most companies lose 50% of their contextual knowledge every 2-3 years due to turnover. Static documentation slows the decay but doesn't stop it. A living knowledge base, fed by actual work product, creates a memory system that persists beyond any individual contributor.
The agent doesn't replace human judgment. It amplifies it — by remembering what we forgot, surfacing what we missed, and keeping our collective knowledge honest about its own freshness.
Try It Yourself
Start small:
- Pick one information stream (GitHub PRs are the easiest)
- Set up a weekly synthesis job
- Share the output in your team channel
- Iterate based on what people actually read
Don't aim for completeness. Aim for usefulness.
What systems have you used to keep team knowledge alive? I'd love to hear what's worked — and what hasn't — in your experience.
