Benchmarks

How this was measured

Everything ran in Docker on one machine, each component pinned to its own cores so nothing competed for CPU.

• Both gateways sit in front of the same upstream with identical model configuration, so neither gets an easier path.
• Each gateway runs as its own documentation recommends: LiteLLM per its production guide, with workers set to the number of vCPUs; Wäg with shipped defaults. Response caching is off on both.
• Closed-loop tests measure maximum throughput. Open-loop tests fire on a fixed schedule and count latency from the intended send time, so a gateway that queues cannot hide the queueing.
• Every response is validated: status, body shape, and for streams the expected chunk count and terminator. Anything else counts as an error, never as a fast response.
• Each scenario runs three times and the tables show the median. With no gateway in the path the same load reaches about 90,000 req/s, far above either gateway, so the results are gateway-bound rather than harness-bound.

Footprint

What each gateway costs before it serves anything: memory at rest, time to ready, image size, and the latency it adds to a single request against an instant upstream.

What it means: Memory decides how many replicas fit on a node, and image size decides how fast a rollout is. At 3.8 MB resident and a 123 MB image, Wäg fits in a sidecar and is ready before a health check times out; 2.3 GB and 11 seconds is a pod you plan around.

Added latency is the gateway's own p50 minus the p50 measured with no gateway in the path.

Under a realistic upstream

A real model answers in hundreds of milliseconds, so thousands of requests are in flight at once. Here the upstream takes 300 ms and requests arrive at a fixed rate whatever the gateway does. At 2,000 req/s roughly 600 requests are in flight at any moment.

What it means: This is the shape of real traffic, where a model answers in hundreds of milliseconds and thousands of requests are in flight. A gateway that queues turns a 300 ms answer into a multi-second one, and nobody notices until peak. Wäg stays at the floor through 2,000 req/s on under a third of its CPU budget. LiteLLM is already queueing at 1,000 and fails 8% of requests at 2,000.

The upstream alone takes 300 ms, so 300 ms is the floor. Everything above it is time spent queueing inside the gateway. The × column always reads the same way: higher means Wäg is ahead. It is Wäg ÷ LiteLLM for throughput and LiteLLM ÷ Wäg for latency.

Throughput and latency

Closed loop against an instant upstream: pure gateway cost, with no model latency to hide behind. These are the numbers a capacity plan needs. They are not the reason to choose a control plane.

4 vCPU, no database

1 vCPU, no database

What it means: This sets how many replicas a traffic level needs, and therefore what the gateway costs to run. The 1 vCPU table is the one to read for Kubernetes: that is the per-core efficiency a pod actually gets.

Streaming, payloads, embeddings, failover

4 vCPU, no database. Streams are 50 tokens 5 ms apart; the soak runs 200 streams per second of roughly 1.2-second answers for three minutes.

What it means: Streaming, long prompts, embeddings and failover are where a production workload spends its time. Read the error columns before the throughput ones: a failed request is a failed user request, not a slow one.

The 64 KB payload figure sits at the harness ceiling: with no gateway in the path the same load reaches 3,711 req/s of 64 KB bodies on one core, so the Wäg number there is a lower bound rather than a maximum.

What this does not show

Every benchmark describes one setup. These are the limits of this one.

• Absolute numbers depend on the host. Both gateways ran on the same machine with pinned cores, so the comparison between them holds, but expect different absolute figures on your own hardware.
• The harness that drove these runs is not published, so you cannot repeat them command for command. The method above is described in enough detail to rebuild an equivalent.
• LiteLLM ships most weeks. These figures were measured against v1.104.2, the current stable release. When they ship a release that changes the forwarding path, we measure it again and publish whatever we find.
• LiteLLM was configured from its production guide, not by its maintainers. A better configuration may exist, and if you find one we will re-run and republish.
• Both gateways ran on one machine with pinned cores. Nothing here measures behaviour across a network, under Kubernetes scheduling, or with noisy neighbours.