Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

Large companies now use models from several vendors at once. Each vendor has its own API format, keys, streaming protocol and prices. If every application integrates every vendor directly, keys spread everywhere, switching models needs code changes, and nobody controls spend. This paper from PayPal describes Cosmos, its unified LLM gateway built on the Envoy proxy. The core split is simple. Slow decisions that wait on other services, such as authentication, budgets and policy, run in a separate service outside the proxy. Fast per-byte work, such as format translation and token counting, runs inside the proxy. As of September 2026 it serves more than 12,000 developers, about 200 billion tokens a day and over 100 models. In a load test it adds only 4 ms of median latency.

Engineers who run model platforms, internal AI infrastructure or cost governance should read it. It shows that the hard part of a gateway is governance, not routing. Identity, budgets and policy carry most of the design weight. Models, prices and policies are stored as validated data instead of code. Keep in mind that this is a production experience report. It has no head-to-head comparison with other gateways, and the latency test covers only the request path. Treat it as an architecture reference.

What problem is it actually solving?

Take an illustrative case. Team A builds a coding assistant on Claude. Team B runs voice support on Gemini. Team C does analytics on OpenAI.

This shows the problem is governance, not forwarding. The authors list ten requirements. A few of them shape the architecture. Clients keep using familiar OpenAI, Anthropic and Gemini APIs. Streaming responses must never be buffered. Access control and data protection fail closed, while monitoring fails open. Adding a model, changing a price or adding a budget must be a data change. Hundreds of live applications cannot be asked to migrate at once.

The architecture

The system has three layers. The data plane is a fleet of autoscaled Envoy proxies, and every request passes through it. A control service sits outside the proxy. It authenticates the caller, looks up the model, checks access and budget, and obtains the vendor credential. If it fails, the request is rejected. Inside the proxy runs a filter written in Rust and compiled to WebAssembly. It translates between vendor formats and counts tokens in the stream. The outer layer is the control plane. Administrators register models, change prices and write policy there. Changes are packaged into versioned bundles and pushed to the data plane.

Why split it this way? The two kinds of work have very different latency profiles. Authentication and policy wait on databases and identity services, so they are I/O-bound. Outside the proxy they cost one call per request. Translation and metering touch every chunk of a streamed reply. One reply can have thousands of chunks. A network hop per chunk would add up quickly.

How one request flows

Say a coding assistant calls Claude using the OpenAI format. The control service first identifies the caller. It then finds the vendor deployment for that model name in the registry. The policy engine checks access and remaining budget. The service obtains the vendor credential, rewrites the path and hands the request back. The proxy passes the body to the translation filter, which converts it to the Anthropic format. As the streamed reply arrives, the filter converts each chunk back to the OpenAI format and keeps a running token count. Headers cannot change during streaming, so usage travels back in trailers. The control service then prices the call and records it.

Translation uses one shared intermediate representation. Each wire format gets one codec that converts to and from this shared form. Five formats need five codecs instead of twenty pairwise converters. If source and target formats match, bytes pass through unchanged. If only the envelope differs, such as Anthropic Messages sent to the Google Cloud version, only envelope fields change. This keeps vendor-specific fields from being lost.

Models and prices are data, not code

In many systems, adding a model or changing a price needs a release. Cosmos puts models, deployments, prices and API capabilities into one registry. The loader validates it strictly. Every format a model advertises must be backed by at least one deployment with both an operation and a converter. Otherwise the gateway refuses to start. The authors' reasoning is direct. Refusing to start is safer than starting and routing requests wrongly.

Prices are stored per million tokens for input, output, cache reads and cache writes. The cost of one request is each token count times its rate, summed. A price change is one data edit that takes effect in the next publication cycle.

Policy and budgets: decide separately from acting

Access and budget rules are written in OPA, an open-source policy engine. The engine only decides. The gateway gathers the data and carries out the effects. This lets the same rule be tested offline and previewed against live data before launch.

The budget design is neat. The gateway keeps only fine-grained usage counters by person, model and time. Every budget is a view over those counters. Suppose a department adds a $50,000 monthly limit in the middle of the month. No data migration is needed. The gateway sums the usage already recorded this month and knows what remains. Rules name organization leaders rather than team lists, so they survive reorganizations.

New rules launch in shadow mode. They first record who would be blocked without blocking anyone. Once verified, they switch to enforcement. Users get a warning before they hit a limit.

Data protection and long-lived voice sessions

Data-loss prevention (DLP) checks for sensitive data before it leaves the company. It is enabled per use case. When a use case's data class requires it, the control service scans prompts for personal and payment data. Policy then decides whether to block, redact or pseudonymize. If this step fails, the request is rejected. For example, a card number in a support ticket is replaced with a placeholder before it goes out.

Real-time voice is another challenge. A call can last 30 minutes over one long-lived connection, while the gateway may deploy, scale or upgrade nodes any day. The authors design around how connections open, stay alive and drain, not just single requests. An old instance stops taking new connections and waits long enough for calls to end. The control service reports itself unavailable before it shuts down. Deployments therefore need no special window, and callers notice nothing.

Results

The authors load-tested the core request path. The upstream was a mock with fixed latency, so any extra time comes from the gateway itself. They used two and four 4-core machines at two load levels, five minutes each. The goal was to see whether overhead changes when load or machine count doubles.

This shows overhead did not change when load doubled or machines were added. That points to a fixed per-request cost rather than queueing. In production, availability is 99.999%. One workload peaked at 60,000 requests per minute. The gateway supports more than 2,000 concurrent voice calls. It was deployed more than ten times during live voice traffic with zero dropped calls.

Coding assistants resend long, stable prefixes every turn: system prompts, tool definitions and history. Vendors allow up to four cache breakpoints in a prompt. Content before a breakpoint is cached and reused when the next turn has the same prefix. The gateway places breakpoints after the last tool definition, the system prompt and the latest message, and requests a one-hour lifetime. The best workloads reach cache hit rates of up to 96%. Cache reads are billed at a discount, so input cost drops with no client changes.

Limitations as I see them

1. The latency test covers only half the path. It measures time until the upstream starts replying and excludes streaming translation. All virtual users shared one credential, so authentication always hit the cache. The load generator ran in the same cluster. Real workloads with more policies and DLP will cost more. Nothing above 2,262 requests per second was tested.
2. There is no controlled comparison. The 200 billion tokens and 99.999% figures are production statistics. Nothing compares this design with other gateways on the same load. The 96% cache hit rate is the best workload only.
3. Centralizing access also centralizes risk. The gateway holds credentials for every vendor, so it is a prime target. Its own access control and change process must meet the bar of the most sensitive workload, and the paper treats this cost only in principle.

How a platform team could try this

1. Wrap, don't rewrite. Put a proxy in front of the existing call path, then move auth, credentials and billing behind it one at a time. No client changes are needed.
2. Split by latency profile. Keep I/O-bound decisions outside the proxy and per-byte work inside it. Write a clear, tested contract between the two.
3. Record usage at the finest grain. Keep raw counters by person, model and time, make budgets views, and run new rules in shadow mode first.
4. Validate configuration strictly. Treat models, prices and policies as data and refuse to start on invalid input.

Bottom line

Cosmos is a thorough design checklist for an enterprise LLM gateway. The two ideas most worth borrowing are "slow decisions outside, fast processing inside" and "every budget is a view."