using all 31 free NVIDIA NIM models at once with automatic routing and failover

been using nvidia NIM free tier for a while and the main annoyance is picking which model to hit and dealing with rate limits (~40 RPM per model).

so i wrote a setup script that generates a LiteLLM proxy config to route across all of them automatically:

• validates which models are actually live on the API
• latency-based routing picks the fastest one each request
• rate limited? retries then routes to next model
• model goes down? 60s cooldown, auto-recovers
• cross-tier fallbacks (coding -> reasoning -> general)

31 models right now - deepseek v3.2, llama 4 maverick/scout, qwen 3.5 397b, kimi k2, devstral 2, nemotron ultra, etc.

5 groups u can target:

• nvidia-auto - all models, fastest wins
• nvidia-coding - kimi k2, qwen3 coder 480b, devstral, codestral
• nvidia-reasoning - deepseek v3.2, qwen 3.5, nemotron ultra
• nvidia-general - llama 4, mistral large, deepseek v3.1
• nvidia-fast - phi 4 mini, r1 distills, mistral small

add groq/cerebras keys too and u get ~140 RPM across 38 models.. all free.

openai compatible so works with any client:

client = openai.OpenAI(base_url="http://localhost:4000", api_key="sk-litellm-master")
resp = client.chat.completions.create(model="nvidia-auto", messages=[...])

setup is just:

pip install -r requirements.txt
python setup.py
litellm --config config.yaml --port 4000

github: https://github.com/rohansx/nvidia-litellm-router

curious if anyone else is stacking free providers like this. also open to suggestions on which models should go in which tier. 🚀