What is an AI gateway?
An AI gateway is one shared entry point between your applications and the AI models they use — OpenAI, Anthropic, Mistral, or models you run yourself. Instead of every application holding its own keys and making its own decisions, every request passes through one place where access, cost, data rules and the record of what happened are handled once, for everyone.
The same three applications, twice
Without a gateway
- Every application holds its own provider keys — and every one of them is a place a key can leak.
- Each team builds its own retries, limits and logging, each slightly differently.
- Spend arrives as one invoice per provider, with no line for the team or feature that caused it.
- Adding or switching a provider means changing code in every application.
With a gateway
- Provider keys live in one place. Applications get their own gateway keys, which can be revoked without touching the provider.
- Retries, failover and caching happen once, the same way, for every application.
- Every request is attributed to a team, an application and a user as it happens.
- Models are added or swapped in configuration. Application code doesn't change.
Five things happen to every request
The details differ between products, but the shape is the same: checks first, the model call in the middle, the record at the end. A request that fails a check stops before anyone is charged for it.
- Check who is asking. The application presents its own gateway key, or a user's company login. Unknown callers are refused. ✗ refused
- Check the limits. Is this team inside its budget and its rate limit? If not, the request stops here — before the provider is contacted. ✗ over budget
- Apply the rules. Organization-wide rules run on the prompt — for example, mask personal data, or keep it from leaving for an external provider. ✗ blocked · ◑ masked
- Pick a model and call it. The gateway chooses which provider and model serve the request, and moves to the next one if a provider is slow or down.
- Record what happened. Who asked, which model answered, what it cost and which rules applied — written down for every request, not reconstructed later.
This is the simplified version. Wäg's full request path has nine stages, including caching and tool policy on agent tool calls: see the request path, stage by stage.
It usually starts with one of these
Nobody buys a gateway because the category exists. They buy one because something on this list happened.
| What you notice | What a gateway does about it | Read more |
|---|---|---|
| The AI bill doubled, and nobody can say which team caused it. | Attributes the cost of every request to a team, an application and a user, and stops spend at a limit you set. | Cost and budgets |
| Provider keys are pasted into a dozen config files and CI secrets. | Holds provider keys in one place. Applications get gateway keys you can scope, rotate and revoke. | Security posture |
| A provider outage took a customer-facing feature down with it. | Retries, and fails over to another provider or model automatically. | Routing and failover |
| Someone asked whether customer data is sent to an external model, and nobody knew. | Applies data rules to every prompt — mask, redact or block — before it leaves. | Policy and guardrails |
| Every team picked its own model, and most of them default to the most expensive one. | Routes each request to the cheapest model that can handle it, by policy rather than by habit. | Intelligent routing |
| An auditor, a customer or the EU AI Act asks what your AI systems actually did. | Keeps a record of every request: who asked, which model answered, what it cost, which rules applied. | EU AI Act and logging |
| Agents now call tools and internal systems with real credentials. | Puts agent tool calls through the same identity, policy and record as model calls. Not every gateway does this — ask. | Agent governance |
Usually one line
Most gateways speak the OpenAI API, which nearly every AI SDK and framework already supports. Adopting one means changing the address your application sends requests to, and the key it sends with them.
client = OpenAI(
- base_url="https://api.openai.com/v1",
- api_key=os.environ["OPENAI_API_KEY"],
+ base_url="https://ai-gateway.internal/v1",
+ api_key=os.environ["GATEWAY_KEY"],
)Prompts, SDK and response handling stay as they are. A good gateway also translates for providers with a different API, such as Anthropic or Gemini, so the application never has to know which provider answered.
The decision most people don't know they're making
A gateway sees every prompt and every response that passes through it. So the first question is not which features it has, but whose infrastructure it runs on.
| Question | Hosted — a vendor runs it | Self-hosted — you run it |
|---|---|---|
| Where prompts and responses go | Through the vendor's network and systems, on the way to the provider | Nowhere new — they stay inside your infrastructure until they reach the provider you chose |
| Who holds your provider keys | The vendor | You |
| Time to the first request | Minutes — sign up and change the base URL | Longer — you deploy it first |
| Who keeps it running | The vendor | Your team, usually the platform team |
| The right choice when | Speed matters most and the data is not sensitive | Data protection, regulation or procurement rules out a third party in the request path |
Not always
You have one application, one provider and one team, and nobody outside engineering is asking about AI spend or data. Calling the provider directly is simpler, and simpler is a feature. Come back when that changes.
Start looking when one of these happens:
- A second team, or a second provider, arrives.
- Someone asks what AI costs per team or per product, and the answer takes a week.
- Data protection or security asks where prompts go.
- An outage at one provider becomes your outage.
- Agents start acting on your systems with real credentials.
When a gateway is not enough
A gateway governs the applications that choose to point at it. Once AI becomes shared infrastructure — many teams, agents with real authority, auditors asking questions — the question changes from how requests are forwarded to who decides what a request may be, for the whole organization. That is the line between an AI gateway and an AI control plane.
An AI gateway you run yourself
Wäg is an open-source AI gateway that runs entirely in your own infrastructure: no hosted version, no vendor in the request path, no phone-home. It speaks the OpenAI API, connects to 25+ OpenAI-compatible providers plus native Anthropic and Gemini, and adds the control-plane layer — identity, hierarchical budgets, policy, agent governance and audit — in the same request path. Apache-2.0, written in Rust.
Already running LiteLLM or another gateway? Wäg imports LiteLLM configuration, and the migration guide covers running both side by side before you switch.
What people ask first
Is an AI gateway the same as an LLM proxy?
Mostly. LLM proxy, AI proxy, LLM gateway and AI gateway all describe one endpoint in front of several model providers. Gateway usually implies more than forwarding — keys, limits, rules and a record — but the terms are not used consistently, so check what a product actually enforces rather than what it is called.
How is it different from an API gateway?
An API gateway manages requests: routing, authentication and rate limits per call. An AI gateway also understands what is inside a model request — tokens, the price per model, streamed responses, prompts that may contain personal data, and providers that can stand in for one another. Some API gateways now add AI features; the useful question is whether they count tokens and cost, or only requests.
Do I have to change my application code?
Usually only the base URL and the key, provided the gateway speaks the OpenAI API and your application uses an OpenAI-compatible SDK or framework — which most do.
Does it slow requests down?
It adds a network hop, so it adds some time. A well-built gateway adds milliseconds, against model responses that take hundreds of milliseconds to several seconds, and a cache hit can make a request much faster than calling the provider at all. Ask any vendor for overhead measured with logging and budgets switched on, not off.
Isn't it a single point of failure?
Only if it runs as a single instance. Production gateways run as several stateless instances behind a load balancer, and failover across providers typically removes more outages than the gateway adds.
Does the gateway vendor see my prompts?
With a hosted gateway, yes: prompts and responses pass through the vendor's systems. With a self-hosted gateway, the vendor is not in the request path at all — the software runs in your infrastructure and talks only to the providers you configure.
Can it route to models we host ourselves?
Yes, if those models expose an OpenAI-compatible API, as vLLM, Ollama and most inference servers do. The gateway treats them like any other provider, so on-premises and cloud models sit behind the same endpoint and the same rules. For models on employees' own computers or inside mobile apps, Wäg Edge records the local calls and reports them to the gateway.
One endpoint. Every model. Complete control.
Run it in your own infrastructure
Everything you need to run a control plane in production is Apache-2.0 and stays that way — routing, HA, caching, failover, budgets, policy, MCP, jobs and observability. Self-hosted, no account, no phone-home.
$ docker run -p 4000:4000 <registry>/waeg-gateway:<version>Not a working pull: the images are not public yet. Design partners receive the registry details privately.
Deploy in 5 minutesShape it as a design partner
Wäg is not publicly released yet. We are working with a small cohort of teams running multi-provider AI in production, and serious about governance and sovereignty, to harden the control plane on real workloads.
- Every Enterprise capability included, at no licence cost
- Hands-on migration — we do it with you, including from LiteLLM
- A direct line to the engineers building it, and real influence on what ships next