AI Navigate
インサイト最新記事一覧AI大全

Are there ways to set up llama-swap so that competing model requests are queued ?

Reddit r/LocalLLaMA / 3/29/2026

💬 OpinionDeveloper Stack & InfrastructureTools & Practical Usage
Read original →
共有:

Key Points

  • The post asks whether LlamaSwap (running on a 48GB workstation) can be configured so that when GPU/host memory is exhausted, incoming model inference requests are queued rather than failing.
  • The author intends to keep using LiteLLM as the front-end API layer while delegating the actual model hosting/swapping behavior to a LlamaSwap instance.
  • It seeks guidance on how to support multiple models behind the same API endpoint so students can request whichever model they want.
  • The author also asks whether using AMD hardware introduces additional complications for LlamaSwap/LiteLLM integration or performance.
  • Overall, the request is focused on operational behavior (request handling and concurrency) and deployment considerations for an educational/student-access setup.

Hello everyone:) as the title says, I am looking to provide a 48gb workstation to students as an API endpoint. I am using litellm currently and want to keep using it but under the hood I would love to get a llama swap instance to run so that I can offer different models and students can just query the one they want. But if no memory is left I would like the job to be queued is there a functionality like that ?

Also I am running on AMD does that introduce any further problems?

submitted by /u/Noxusequal
[link] [comments]

関連おすすめサービス

※当サイトはアフィリエイト広告を利用しています
🎧

Notta搭載AI議事録イヤホン ZENCHORD1

AI時代の仕事術。Notta搭載で会議の議事録を自動生成するスマートイヤホン。

🎙️

AI搭載ボイスレコーダー Plaud

世界100万人が愛用。AIで文字起こし・要約を自動化するボイスレコーダー。

🖼️

画像高画質化AIツール Aiarty Image Enhancer

AIで画像を高画質化。写真・イラストを簡単にアップスケール。