Getting Started with Tokens — The Unit Behind Every LLM
Zenn / 2026/4/29
💬 オピニオンIdeas & Deep Analysis
要点
- LLMの基礎単位であるトークンを軸に、BPEやサブワードから文脈ウィンドウ、prefillとdecodeの違いまでをベンダー非依存で整理する。
A definitive, vendor-neutral field guide to the unit that drives every LLM bill, every latency budget, and every context-window error. From subwords and BPE to context windows, prefill vs decode, prompt anatomy, hygiene principles, measurement, anti-patterns, the strategic case for optimization, and the seven technique families that organize all known optimizations. 16 main chapters plus 2 appendices and a closing column (20 pieces total, including the preface). Written from first principles, deliberately free of vendor prices and model-version specifics so the mental model survives the next price sheet.
