LiteLLM musings
Why do I have LiteLLM?
- The idea that I need to swap harnesses to use different models annoyed me
- That each harness would need to be configured and managed added to this annoyance
- Claude Code, Github Copilot VSCode, OMP, Github Copilot CLI, Vibe CLI, Antigravity CLI all felt overly bloated and i liked the configurability of Pi.
- I needed a solution to handle the many different MCP server without manually managing a bunch of seperate
.mcp.jsonconfig files. - I needed a solution where the Pi harness could swap between locally hosted models and cloud models
- Phoenix and Prometheus can just call
/metricsfor everything.. that part is nice.
The Pros
- LiteLLM isn't just a model registry, it is an agent registry too.
- So a model can be abstracted away behind an agent name (allows for swapping a backend model without impacting downstream dependencies)
- When calling
/modelin Pi, you see things likefactory-revieweralongsideChaptGPT Luna 6 - LiteLLM is also a MCP registry so you can have specific MCP servers associated with only specific agent profiles (!!!) which is incredibly useful in preventing context bloat.
- My local machines and kubernete cluster independent software factory pods can all use the same middleware proxy. I have thus minimized the number of configurations I need to manage
The Cons
- Llama.cpp + LiteLLM + Pi together aren't great about switching "thinking levels" on models so each thinking mode needs to be declared as entirely seperate model in llama.cpp and LiteLLM. This causes a model reloaded on local models if I ever switch their thinking levels which is a massive slowdown.
- Still haven't gotten the skill registry working
- Wondering whether an MCP registry is even the best solution for managing MCP access and MCP bloat issues.
- There are other MCP's who act as search retrieval for the rest of the MCP's... theoretically that could be the mechanisms for handling identity and access while managing MCP context bloat.
- There are some quiet failures that are probably my fault and need to address
Config file Samples
model_list:
model_list:
- model_name: factory-tdd # tests are mechanical; don't think
litellm_params:
model: openai/qwen3.8-27b-medium
api_base: http://127.0.0.1:11437/v1 # AMD
extra_body:
chat_template_kwargs:
enable_thinking: false
- model_name: factory-dev # agentic loop; thinking pays, xhigh stalls
litellm_params:
model: openai/qwen3.8-27b-medium
api_base: http://127.0.0.1:11437/v1
- model_name: factory-reviewer # a DIFFERENT FAMILY from the authors
litellm_params:
model: openai/gemma-4-26b
api_base: http://127.0.0.1:11441/v1 # NVIDIA
- model_name: DeepseekFlash
litellm_params:
model: deepseek/deepseek-flash
api_key: os.environ/DEEPSEEK_API_KEY
rpm: 30
Agents, their models, and their MCP servers.
| Stage key | Model alias | MCP servers |
|---|---|---|
factory-po |
factory-po (qwen @AMD) |
deepwiki, context7, memory |
factory-tdd |
factory-tdd (qwen @AMD) |
context7, memory |
factory-dev |
factory-dev (qwen @AMD) |
memory |
factory-devops |
factory-devops (qwen @AMD) |
kubernetes, prometheus, memory |
factory-reviewer |
factory-reviewer (gemma @NVIDIA) |
memory |
factory-advisor |
factory-advisor (gemma @NVIDIA) |
(none — read-only) |
Good stuff so far. Plan to continue with it.