LiteLLM musings

Why do I have LiteLLM?

  • The idea that I need to swap harnesses to use different models annoyed me
  • That each harness would need to be configured and managed added to this annoyance
  • Claude Code, Github Copilot VSCode, OMP, Github Copilot CLI, Vibe CLI, Antigravity CLI all felt overly bloated and i liked the configurability of Pi.
  • I needed a solution to handle the many different MCP server without manually managing a bunch of seperate .mcp.json config files.
  • I needed a solution where the Pi harness could swap between locally hosted models and cloud models
  • Phoenix and Prometheus can just call /metrics for everything.. that part is nice.

The Pros

  • LiteLLM isn't just a model registry, it is an agent registry too.
  • So a model can be abstracted away behind an agent name (allows for swapping a backend model without impacting downstream dependencies)
  • When calling /model in Pi, you see things like factory-reviewer alongside ChaptGPT Luna 6
  • LiteLLM is also a MCP registry so you can have specific MCP servers associated with only specific agent profiles (!!!) which is incredibly useful in preventing context bloat.
  • My local machines and kubernete cluster independent software factory pods can all use the same middleware proxy. I have thus minimized the number of configurations I need to manage

The Cons

  • Llama.cpp + LiteLLM + Pi together aren't great about switching "thinking levels" on models so each thinking mode needs to be declared as entirely seperate model in llama.cpp and LiteLLM. This causes a model reloaded on local models if I ever switch their thinking levels which is a massive slowdown.
  • Still haven't gotten the skill registry working
  • Wondering whether an MCP registry is even the best solution for managing MCP access and MCP bloat issues.
  • There are other MCP's who act as search retrieval for the rest of the MCP's... theoretically that could be the mechanisms for handling identity and access while managing MCP context bloat.
  • There are some quiet failures that are probably my fault and need to address

Config file Samples

model_list:

model_list:
  - model_name: factory-tdd # tests are mechanical; don't think
    litellm_params:
      model: openai/qwen3.8-27b-medium
      api_base: http://127.0.0.1:11437/v1 # AMD
      extra_body:
        chat_template_kwargs:
          enable_thinking: false

  - model_name: factory-dev # agentic loop; thinking pays, xhigh stalls
    litellm_params:
      model: openai/qwen3.8-27b-medium
      api_base: http://127.0.0.1:11437/v1

  - model_name: factory-reviewer # a DIFFERENT FAMILY from the authors
    litellm_params:
      model: openai/gemma-4-26b
      api_base: http://127.0.0.1:11441/v1 # NVIDIA

  - model_name: DeepseekFlash
    litellm_params:
      model: deepseek/deepseek-flash
      api_key: os.environ/DEEPSEEK_API_KEY
      rpm: 30

Agents, their models, and their MCP servers.

Stage key Model alias MCP servers
factory-po factory-po (qwen @AMD) deepwiki, context7, memory
factory-tdd factory-tdd (qwen @AMD) context7, memory
factory-dev factory-dev (qwen @AMD) memory
factory-devops factory-devops (qwen @AMD) kubernetes, prometheus, memory
factory-reviewer factory-reviewer (gemma @NVIDIA) memory
factory-advisor factory-advisor (gemma @NVIDIA) (none — read-only)

Good stuff so far. Plan to continue with it.

← All posts