From LiteLLM to Wäg in an afternoon, not a quarter
LiteLLM gets teams to multi-provider fast — that is why so many production stacks run it. What it does not give you is a governance layer: hierarchical budgets that stop spend before the provider is called, policy enforced in the request path, tool calls held for a named approver, and an audit record you can hand to a compliance team. Wäg takes the place of the LiteLLM proxy and adds that layer. You keep your SDK, your model aliases and your application code. You change one base URL.
What changes
- The base URL your clients point at
- Where configuration lives
- The proxy process you operate
What doesn't
- Your SDK — openai, langchain, llamaindex, the LiteLLM SDK as a client, anything OpenAI-compatible
- The model aliases your code references
- Request and response shapes
- Streaming behaviour, tool calling, vision
- Your provider accounts and API keys
Four steps, none of them a rewrite
1 · Import your configuration
Reads your LiteLLM config.yaml — on the command line, or pasted under Platform → System → Import in the console — and produces Wäg configuration: models and aliases, provider credential references, routing strategy, fallbacks, Redis, projects as applications with budgets, virtual keys, MCP servers and A2A agents.
$ waeg import litellm --from config.yaml -o waeg.yaml$ waeg config validate --config waeg.yamlWhat to check is the import report. Every key ends up in one of four lists — mapped, skipped, needs_human and notes — and the command line prints the counts with samples:
mapped
model_list (12 rows) → 6 aliases: gpt-4o, claude, fast, cheap, embed, vision
model_list[3].litellm_params.rpm: 600 → relative deployment weight
litellm_settings.fallbacks → alias fallbacks
litellm_settings.num_retries: 3 → routing.max_retries: 3
router_settings.routing_strategy → routing.strategy: weighted (see notes)
projects[0] (max_budget, budget_duration) → application app_ll_search + application budget
skipped
litellm_settings.success_callback → re-create under Observability
needs_human
general_settings.master_key: os.environ/LITELLM_MASTER_KEY → set WAEG_MASTER_KEY
litellm_settings.cache_params → re-create under Caching
notes
routing_strategy "latency-based-routing-v2" is not known → weightedIllustrative output. The point of showing it is that the report is a real artifact: nothing fails silently, anything without a Wäg equivalent is listed for review rather than approximated, and secrets are never echoed back.
2 · Point one base URL
Wäg serves the OpenAI-compatible surface — /v1/chat/completions, /v1/responses (stateless), /v1/embeddings, /v1/images/*, /v1/audio/*, /v1/videos, /v1/rerank, /v1/ocr, /v1/moderations, /v1/files, /v1/batches, /v1/realtime, /v1/models — plus Anthropic's /v1/messages, Gemini's /v1beta, and /v1/mcp and /v1/a2a for agents. LiteLLM clients such as Cline and Roo keep working: GET /model/info answers with or without /v1, x-litellm-api-key is accepted on pass-through and MCP routes, and x-litellm-session-id groups calls into conversations. Streaming is true SSE.
client = OpenAI(
- base_url="http://litellm:4000/v1",
+ base_url="https://waeg.internal/v1",
api_key=os.environ["GATEWAY_KEY"],
)What to check: GET /v1/models returns your aliases, then one streaming and one tool-calling request per alias.
3 · Verify in parallel
Run both. Send a percentage of traffic to Wäg, or shadow production traffic to it, and compare. Because replicas are stateless and share their counters through Redis and their configuration through Postgres, you can scale the canary up and down without losing budget state.
- Routing decisions — x-waeg-routed-model on every response, x-waeg-route-reason with intelligent routing
- Cache behaviour — x-waeg-cache, and whether hit rates match your expectations
- Token counts and cost attribution against your provider invoice
- p50 / p95 / p99 latency, error rates and fallback rates
- Compatibility signals — request and response fields your clients rely on that the gateway forwarded or dropped, counted from your own traffic
4 · Cut over
Move the base URL for one service, then the rest. Keep the LiteLLM deployment running until you are satisfied — rolling back is the same one-line change in reverse.
A reasonable order: internal tools first, then batch and offline workloads, then one production service, then everything.
Change the base URL back. That is the whole procedure — Wäg holds no state your applications depend on, and your provider accounts, keys and model aliases are untouched. Keep both deployments running through the transition. There is no lock-in step in this migration, by design.
Config field by config field
| LiteLLM | Wäg | Notes |
|---|---|---|
model_list[].model_name | model alias | Preserved verbatim — your code keeps working |
litellm_params.model | deployment: provider + model_id | Several rows with the same model_name become weighted deployments of one alias |
api_key: os.environ/X | api_key_env: X | Read from the environment at runtime; switch to a vault reference later if you prefer |
litellm_params.api_base | provider base_url | Another host or region becomes another provider |
router_settings.routing_strategy | routing.strategy | simple-shuffle and usage-based → weighted, least-busy and latency-based → latency, cost-based → cost; anything else → weighted, with a note. Wäg adds intelligent |
litellm_settings.fallbacks | alias fallbacks | Ordered stronger-first on the intelligent strategy |
litellm_settings.num_retries | routing.max_retries | Plus a circuit breaker per deployment |
general_settings.master_key | master_key | An os.environ reference is listed for review: set WAEG_MASTER_KEY. Plus virtual keys with budgets and model allow-lists |
projects[]: max_budget / budget_duration | application + budget | Wäg is hierarchical: org → team → user → application → model → key |
rpm on a deployment | deployment weight | A relative weight, not a rate limit. A project's rpm_limit and tpm_limit go to its application budget policy; per-deployment TPM does not map |
cache_params | re-create after import | Under Caching: exact, canonical, near-dup and semantic tiers with anti-false-hit gates |
guardrails | re-create after import | The same vendors exist as catalog items, in an ordered chain: pre-call, mid-stream and post-call |
success_callback / failure_callback | re-create after import | Under Observability: OTLP, Langfuse and HTTP callbacks, plus Prometheus metrics |
custom Python callbacks | — no equivalent | Wäg is not a Python process. Use an HTTP webhook policy engine instead |
keys[] with a project_id | virtual keys | Bound to the project's application, so a coding agent's budget rides on its key |
mcp_servers | platform MCP servers | Bearer and API-key auth become static credentials; user OAuth and token exchange need Enterprise. Imported tools are pinned |
agents[] | platform A2A agents | Refresh each agent card after the import; the gateway fetches and checks it |
import litellm (in-process) | a client of the gateway | The SDK keeps working pointed at Wäg; its in-process routing and callbacks do not |
Including what LiteLLM does that Wäg doesn't
Check these against your setup before you start. If any of the first two are load-bearing for you, migration is a real project rather than an afternoon, and you should know that now rather than halfway through.
No in-process routing
LiteLLM can be imported as a Python library. Pointed at Wäg, the SDK keeps working as a client, but its in-process routing and callbacks do not carry over — routing happens in the gateway instead.
No custom Python callbacks
LiteLLM lets you drop arbitrary Python into the request lifecycle. Wäg's extension point is an HTTP webhook policy engine — cleaner to operate and secure, less immediately flexible. Heavy callback logic becomes a small service.
Fewer exotic providers, more faithful ones
LiteLLM's provider coverage is broader. Wäg has about a dozen native provider kinds, plus any OpenAI-compatible server, with native Anthropic and Gemini adapters that preserve tool, vision and structured-output fidelity rather than flattening everything through one shape. Check your list first.
Different routing philosophy
Wäg scores requests structurally — tools, vision, context length, reasoning cues — into hard capability floors, then picks the cheapest model that clears them. Deterministic and explainable in headers. No ML router in the hot path.
Different cache philosophy
Four tiers — exact, canonical, near-dup, semantic — with anti-false-hit gates. "Capital of France" and "capital of Spain" are a miss. Numbers, negation and entity changes break a hit.
Governance is hierarchical, not flat
This is the actual reason to migrate. LiteLLM's budgets are per-key. Wäg's stack across org → team → user → application → model → key with fair-share caps, so the model matches how organizations are actually shaped.
What you gained
Budgets that stop spend first
Stacked limits across org → team → user → application → model → key. Every request is checked against every counter that applies; the first breach rejects the call before provider spend. With WAEG_CLUSTER=1 they also fail closed if Redis is unreachable.
Policy in the request path
An ordered scanner chain pre-call, mid-stream on a rolling window, and post-call. PII, injection, moderation, or your own webhook. Block, mask or redact per rule.
Governed agent loops
Remote MCP servers get per-server credentials — static key, forwarded user token or OAuth client credentials, and with Enterprise the user's own OAuth connection or token exchange — plus tool allow-lists and pinning. Tool calls can wait for a named approver, in Slack or Teams with Enterprise, and every hop is budgeted and logged.
An audit record
Every request, routing decision, policy verdict and cost in request logs, OTLP traces and Prometheus metrics — and, once turned on, a signed, hash-chained evidence ledger anyone can check offline with waeg ledger verify. Enterprise adds evidence packs mapped to the AI Act and DORA, with gaps flagged, and AI content marking.
One binary
Rust, distroless, non-root. No shell, interpreter or package manager in the gateway image, signed releases with an SBOM, and no Node runtime in the gateway — the optional console assistant runs as its own service.
An operator to run it
A first-party Kubernetes operator declares the gateway — topology, storage, OpenFGA, HPA or KEDA scaling — and reconciles providers, models, budgets, keys and guardrails as resources. Misconfigurations show up as conditions with a reason, not as a pod that fails an hour later.
See where the gateway ends and the control plane begins · See the benchmarks
Questions that come up
Do I have to change my application code?
No. You change the base URL your OpenAI-compatible client already uses.
Does my LiteLLM config port over?
Most of it, via waeg import litellm or Platform → System → Import in the console. You get a report of what mapped, what was skipped and what needs a human, and nothing fails silently.
Can I run both at once?
Yes, and it is the recommended path. Shadow or split traffic and compare routing decisions, cost and latency before cutting over.
Is Wäg a fork of LiteLLM?
No. It is an independent implementation in Rust, built around an inference-request model rather than provider SDK dispatch. The compatibility is deliberate; the lineage is separate.
Does it support streaming?
Yes — true server-sent events, not buffered-then-chunked.
Tool calling and vision?
Yes, across OpenAI, Anthropic and Gemini request shapes. Native adapters preserve provider-specific fidelity rather than flattening to a lowest common denominator.
What about Anthropic's and Gemini's native APIs?
Supported as first-class request shapes, not only as translations of the OpenAI shape.
Do I need Redis?
For more than one replica, yes — it keeps budgets and rate limits consistent across replicas, next to the Postgres and OpenFGA that production needs anyway. Embedded mode needs nothing external, but it is single-node and keeps its state in memory.
Does it run on Kubernetes?
Yes, two ways. A first-party operator declares the gateway and its catalog as resources — stateless replicas, HPA or KEDA scaling, shared budget counters, health and readiness probes — and a Helm chart is there if you prefer to run it yourself.
Is it really Apache-2.0?
Yes. The gateway's production path is Apache-2.0: routing, HA, caching, failover, budgets, guardrails, MCP, jobs, observability and the evidence ledger. Enterprise adds identity (SSO, SCIM, IdP tokens), audit and evidence packs, agent payments, Slack and Teams approvals, pass-through guardrails and support.
Does it phone home?
No. No telemetry and no analytics, and the licence is a signed offline token with no licence server. It runs fully air-gapped: watch its outbound connections and you see only the providers you configured.
What if we need help migrating?
Design partners get hands-on migration support, including from LiteLLM — we do it with you rather than handing over a guide.
One endpoint. Every model. Complete control.
Run it in your own infrastructure
Everything you need to run a control plane in production is Apache-2.0 and stays that way — routing, HA, caching, failover, budgets, policy, MCP, jobs and observability. Self-hosted, no account, no phone-home.
$ docker run -p 4000:4000 <registry>/waeg-gateway:<version>Not a working pull: the images are not public yet. Design partners receive the registry details privately.
Deploy in 5 minutesShape it as a design partner
Wäg is not publicly released yet. We are working with a small cohort of teams running multi-provider AI in production, and serious about governance and sovereignty, to harden the control plane on real workloads.
- Every Enterprise capability included, at no licence cost
- Hands-on migration — we do it with you, including from LiteLLM
- A direct line to the engineers building it, and real influence on what ships next