AI Navigate
インサイト最新記事AI大全カオスマップ

Karpathy's MicroGPT running at 50,000 tps on an FPGA

Reddit r/LocalLLaMA / 5/3/2026

📰 NewsDeveloper Stack & InfrastructureSignals & Early TrendsTools & Practical UsageModels & Research
Read original →
共有:

Key Points

  • Karpathy’s MicroGPT is reportedly running at around 50,000 tokens per second on an FPGA using a very small model with just 4,192 parameters.
  • The write-up emphasizes that much of the throughput comes from keeping model weights on the chip (onboard ROM) instead of fetching them from external memory.
  • The post notes a practical limitation: with current FPGAs and 16-bit weights, onboard ROM caps the model size at roughly 20–30 million parameters.
  • It suggests that future increases in onboard ROM capacity—or FPGAs specialized for small language models (SLMs)—could enable larger models to achieve similar high inference speeds.
  • Project details and the related repository are provided for readers to inspect and reproduce the approach.

Sure, it's only 4,192 parameters, but it's a start. Project write-up here: https://v2.talos.wtf/ and github repository here: https://github.com/Luthiraa/TALOS-V2

Some of the speed comes from having the weights onboard, rather than in external memory. Onboard ROM means with 16 bit weights current FPGAs max out at 20-30 million parameters, but maybe this and Taalas (https://taalas.com/ - similar names are unlikely a coincidence) will lead to more onboard ROM appearing in FPGAs or FPGAs dedicated to SLMs.

submitted by /u/jawondo
[link] [comments]

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。