AI Navigate
アップデート最新記事AI大全カオスマップ

Anyone running Kimi on low VRAM + offloading to RAM? (im sure most)

Reddit r/LocalLLaMA / 5/5/2026

💬 OpinionDeveloper Stack & InfrastructureTools & Practical Usage
Read original →
共有:

Key Points

  • A Reddit user tests running the Kimi model with limited GPU memory (e.g., a 12GB Tesla T4) by offloading the rest of the model to system RAM, aiming to understand the performance impact.
  • They report throughput results on a dual Xeon Platinum CPU setup (48 cores, 1.5TB RAM) with CPU-only execution reaching about 20 tokens/s input and 1.6 tokens/s output, described as very poor.
  • They mention using NUMA and observe an unexpected behavior: a Q8 model (from Unsloth) runs slightly faster than a Q4 model on their system.
  • The post focuses on practical benchmarking and questions about how quantization level and RAM offloading affect output token speed on low-VRAM hardware.
  • Overall, it highlights the performance trade-offs and tuning considerations for local LLM inference when VRAM is insufficient.

Im curious how much output token benefits from something smaller like a 12gb Tesla T4, and offloading the remainder of the model to RAM

I get about ~1.6t/s output ~20t/s input CPU only.. which is obviously terrible. I'm using NUMA.. I have dual xeon platinum 24c(so 48c/96t) and 1.5T of RAM

Strangely enough, the Q8 model from un sloth, run slightly faster than the Q4 model on my system

submitted by /u/Creative-Type9411
[link] [comments]

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。