Skip to content

LLM Gateway Proxy#

An LLM gateway proxy design sits between an application and the dozen LLM providers it might call (OpenAI, Anthropic, Bedrock, Vertex, self-hosted). It speaks one unified API, routes each request to the right model, enforces tenant rate limits and budgets, caches semantically similar prompts, and records token usage for billing and debugging. Products like Portkey, Helicone, and LiteLLM live in this niche.

Problem statement (interviewer prompt)

Design an LLM gateway that exposes a single OpenAI-compatible API in front of multiple LLM providers. Support routing (failover, load-balance, cost-aware), per-tenant rate limits, a semantic cache, an encrypted key vault for provider credentials, and detailed observability (token usage, latency, errors). Target 50k req/s peak, sub-50 ms gateway overhead, 99.95% availability.

flowchart LR
  APP([App / SDK<br/>OpenAI-compatible])
  LB[Anycast LB]
  GW[Gateway<br/>auth, rate limit, routing]
  CACHE[(Semantic cache<br/>embeddings + key-value)]
  VAULT[(Key vault<br/>provider creds)]
  ROUTE[Router<br/>cost, latency, health]
  OAI[OpenAI]
  ANT[Anthropic]
  BED[Bedrock]
  VTX[Vertex]
  SELF[Self-hosted vLLM]
  OBS[Observability<br/>usage, traces]

  APP --> LB
  LB --> GW
  GW --> CACHE
  GW --> VAULT
  GW --> ROUTE
  ROUTE --> OAI
  ROUTE --> ANT
  ROUTE --> BED
  ROUTE --> VTX
  ROUTE --> SELF
  GW --> OBS

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP client;
    class LB,GW edge;
    class ROUTE service;
    class CACHE cache;
    class VAULT datastore;
    class OAI,ANT,BED,VTX,SELF external;
    class OBS obs;

The translation layer is unglamorous but critical. Every provider speaks a slightly different dialect: OpenAI tools vs Anthropic tools, Bedrock Converse vs native APIs, streaming SSE vs Server-Sent vs Vertex websockets. The gateway normalizes everything to one schema in, one schema out, and silently rewrites both sides. Bugs here are subtle: a misformatted tool result shape can make a frontier model start hallucinating.

Routing is the cost lever. A request can be served by the model the user asked for, by a cheaper equivalent, by a fallback if the primary is down, or by a cached response. The router blends a static policy (this tenant always uses Anthropic) with live signals (this provider's p95 is up 3x in the last minute) and a cost model (output token price differs 20x across providers). Sticky routing keeps a multi-turn conversation on the same provider so prompt caches stay warm.

Observability and key management justify the gateway. Even a tiny shop benefits from a central place that sees every LLM call, signs requests with the right key for each provider, redacts secrets in logs, and produces a per-tenant token usage report. The semantic cache is the cherry on top: identical or near-identical prompts return in milliseconds at zero LLM cost.

Problem statement (interviewer prompt)

Design an LLM gateway in the style of Portkey, Helicone, or LiteLLM. It exposes one OpenAI-compatible API across OpenAI, Anthropic, Bedrock, Vertex, and self-hosted vLLM. Features: per-tenant API keys and rate limits, routing policies (failover, load-balance, cost-aware), semantic cache, encrypted provider credentials, full request/response logging, token-level observability, and prompt versioning. Targets: 50k req/s peak, sub-50 ms gateway overhead at p99, 99.95% availability, multi-region deployment.

Requirements#

Functional

  • OpenAI-compatible REST API (/v1/chat/completions, /v1/embeddings, /v1/messages).
  • Streaming (SSE) end-to-end across providers that support it.
  • Routing strategies: static, weighted, cost-aware, latency-aware, failover chain.
  • Per-tenant API keys, separate from provider keys held in the vault.
  • Per-tenant rate limits (requests/min, tokens/min, dollars/day).
  • Semantic cache with embedding-based lookup; exact-match cache as a fast path.
  • Prompt template registry with versioning and A/B routing.
  • Full request/response logs with PII redaction.
  • Token usage and cost analytics per tenant, per model, per route.

Non-functional

  • p50 gateway overhead under 10 ms; p99 under 50 ms (excluding provider RTT).
  • 50k req/s peak across regions.
  • 99.95% availability.
  • No customer-provider keys at rest in plaintext.
  • Hot-reloadable routing config (no deploy for routing changes).

Top-level architecture#

flowchart TB
  subgraph Client[Clients]
    SDK([OpenAI SDK])
    APP([Custom app])
  end

  subgraph Edge[Edge tier]
    DNS[GeoDNS]
    ALB[Anycast LB]
    GW[Gateway pod<br/>stateless, autoscaled]
  end

  subgraph Auth[Auth + policy]
    AUTH[API key auth]
    QUOTA[Quota service<br/>Redis token buckets]
    POL[Policy engine<br/>routing, redaction]
  end

  subgraph Caches[Caches]
    EXACT[(Exact cache<br/>Redis, key=hash)]
    SEM[(Semantic cache<br/>vector DB + Redis)]
    EMB[Embedding service<br/>small model]
  end

  subgraph Core[Routing core]
    ROUTE[Router<br/>policy + signals]
    HEALTH[Health board<br/>per-provider p95]
    COST[Cost model<br/>per-model pricing]
    ADAPT[Provider adapters<br/>OpenAI, Anthropic, Bedrock, Vertex, vLLM]
  end

  subgraph Sec[Security]
    VAULT[(KMS-backed key vault)]
    REDACT[PII redactor]
  end

  subgraph Data[Data plane]
    LOGS[(Request log<br/>S3 + ClickHouse)]
    USE[(Usage ledger<br/>per-tenant rollups)]
    PROMPT[(Prompt registry<br/>Postgres)]
    EVENTS[[Kafka<br/>events for billing + obs]]
  end

  subgraph Prov[Providers]
    OAI[OpenAI]
    ANT[Anthropic]
    BED[Bedrock]
    VTX[Vertex]
    SELF[Self-hosted vLLM]
  end

  SDK --> DNS
  APP --> DNS
  DNS --> ALB
  ALB --> GW
  GW --> AUTH
  GW --> QUOTA
  GW --> POL
  GW --> EXACT
  EXACT -. miss .-> EMB
  EMB --> SEM
  SEM -. miss .-> ROUTE
  ROUTE --> HEALTH
  ROUTE --> COST
  ROUTE --> ADAPT
  ADAPT --> VAULT
  ADAPT --> OAI
  ADAPT --> ANT
  ADAPT --> BED
  ADAPT --> VTX
  ADAPT --> SELF
  GW --> REDACT
  GW -.events.-> EVENTS
  EVENTS --> LOGS
  EVENTS --> USE
  POL --> PROMPT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class SDK,APP client;
    class DNS,ALB,GW edge;
    class AUTH,QUOTA,POL,ROUTE,HEALTH,COST,ADAPT,REDACT,EMB service;
    class EXACT,SEM cache;
    class VAULT,LOGS,USE,PROMPT datastore;
    class EVENTS queue;
    class OAI,ANT,BED,VTX,SELF external;

Data model#

tenants(
  id uuid PK,
  name text,
  plan text,
  monthly_budget_cents bigint,
  created_at timestamptz
);

api_keys(
  id uuid PK,
  tenant_id uuid FK,
  hashed_key text UNIQUE,
  prefix text,            -- first 8 chars, shown in UI
  scopes text[],
  created_at timestamptz,
  revoked_at timestamptz
);

provider_credentials(
  id uuid PK,
  tenant_id uuid FK,
  provider text,          -- openai | anthropic | bedrock | vertex | self
  vault_ref text,         -- pointer to KMS-encrypted blob
  region text,
  created_at timestamptz
);

routing_policies(
  id uuid PK,
  tenant_id uuid FK,
  name text,
  rules jsonb,            -- ordered list of {match, action} rules
  version int,
  active bool
);

prompt_templates(
  id uuid PK,
  tenant_id uuid FK,
  slug text,
  version int,
  body text,
  variables jsonb,
  active bool,
  created_at timestamptz,
  UNIQUE(tenant_id, slug, version)
);

usage_event(
  ts DateTime64,
  tenant_id String,
  api_key_id String,
  provider String,
  model String,
  route String,
  tokens_in UInt32,
  tokens_out UInt32,
  latency_ms UInt32,
  cache_hit Enum8('miss', 'exact', 'semantic'),
  cost_micros UInt64,
  status_code UInt16
);

Component deep-dive#

Provider adapters#

The adapter layer is where complexity hides. Each provider gets its own adapter that translates the unified request to the provider's native format, signs with the correct credentials, handles streaming, and translates the response back.

flowchart LR
  REQ[Unified request<br/>OpenAI schema]
  ADAPT{Provider}
  OAI[OpenAI adapter<br/>passthrough]
  ANT[Anthropic adapter<br/>messages, tools]
  BED[Bedrock adapter<br/>Converse API]
  VTX[Vertex adapter<br/>generateContent]
  SELF[vLLM adapter<br/>OpenAI-compatible]
  SIGN[Sign with vault key]
  STREAM[Stream parser<br/>SSE / chunks]
  NORM[Normalize response]
  OUT[Unified response]

  REQ --> ADAPT
  ADAPT --> OAI
  ADAPT --> ANT
  ADAPT --> BED
  ADAPT --> VTX
  ADAPT --> SELF
  OAI --> SIGN
  ANT --> SIGN
  BED --> SIGN
  VTX --> SIGN
  SELF --> SIGN
  SIGN --> STREAM
  STREAM --> NORM
  NORM --> OUT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class REQ,OUT service;
    class ADAPT,SIGN,STREAM,NORM compute;
    class OAI,ANT,BED,VTX,SELF external;

Adapters must preserve streaming. The gateway proxies SSE chunks as they arrive, with a tiny in-memory buffer to count tokens. Closing the upstream connection cleanly when the client disconnects matters: an LLM call left running after the client gave up still costs money.

Router#

The router picks a provider for each request. It receives a request, a tenant's policy, live health signals, and the cost model, and returns an ordered list of (provider, model, key) tuples to try.

flowchart TB
  IN[Request + tenant policy]
  PARSE[Parse rules]
  MATCH{Match rule}
  STATIC[Static route]
  WEIGHT[Weighted load-balance]
  COST[Cost-aware: cheapest healthy]
  LAT[Latency-aware: lowest p95]
  FB[Failover chain]
  RANK[Rank by signals]
  STICKY{Existing session?}
  PIN[Pin to last provider]
  OUT[Ordered try-list]

  IN --> PARSE
  PARSE --> MATCH
  MATCH --> STATIC
  MATCH --> WEIGHT
  MATCH --> COST
  MATCH --> LAT
  MATCH --> FB
  STATIC --> RANK
  WEIGHT --> RANK
  COST --> RANK
  LAT --> RANK
  FB --> RANK
  RANK --> STICKY
  STICKY -- yes --> PIN
  STICKY -- no --> OUT
  PIN --> OUT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class IN,OUT service;
    class PARSE,MATCH,STATIC,WEIGHT,COST,LAT,FB,RANK,STICKY,PIN compute;

The health board is a moving window of per-provider p95 latency, 5xx rate, and 429 rate, refreshed every few seconds via a sidecar that scrapes recent traces. A provider that hits a 429 burst gets temporarily down-weighted, not eliminated, so traffic flows back as soon as it recovers.

Semantic cache#

Exact-match cache is a hash of (model, normalized prompt, parameters) into Redis. Semantic cache embeds the prompt with a cheap model and looks up nearest neighbors in a vector store; a hit must satisfy a similarity threshold and a "no parameters drift" check.

sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant E as Exact cache
  participant V as Vector cache
  participant EM as Embedding service
  participant L as LLM provider

  C->>G: chat completion
  G->>E: lookup hash
  E-->>G: miss
  G->>EM: embed prompt
  EM-->>G: vector
  G->>V: ANN query, threshold 0.97
  V-->>G: hit (cached response)
  G-->>C: response (cache_hit=semantic)

  Note over G,L: Miss path
  G->>L: stream completion
  L-->>G: tokens
  G-->>C: stream
  G->>E: store hash -> response
  G->>V: store vector -> response

Semantic cache only fires when the response is deterministic-friendly: temperature=0 or low, no tool calls, no streaming subscribers waiting on a session. Tool-using turns must not be cached because the next turn depends on tool outputs that were specific to that prior call.

Key vault and security#

Provider credentials never live in the gateway pod's environment or memory longer than a request. The gateway calls the vault per-request (or per-tenant short-lived lease) and discards the key on response. The vault encrypts at rest with a KMS, audits every fetch, and supports per-credential rotation.

PII redaction runs before request logging. A regex + named-entity model strips emails, phone numbers, and credit card numbers into placeholders. The original prompt is still sent to the provider, only the log is redacted, and a customer-controlled toggle can disable logging entirely for HIPAA tenants.

Rate limit and quota#

Three layers of limits stack:

  1. Global gateway capacity (defends infrastructure).
  2. Per-tenant rate (requests/min, tokens/min) using sliding window in Redis.
  3. Per-tenant budget (cents/day) tracked in the usage ledger.
flowchart LR
  REQ[Request]
  G[Global cap<br/>token bucket]
  T[Tenant rate<br/>Redis sliding window]
  B[Tenant budget<br/>daily ledger]
  PASS[Allow]
  DENY[429 / 402]

  REQ --> G
  G -->|under| T
  G -->|over| DENY
  T -->|under| B
  T -->|over| DENY
  B -->|under| PASS
  B -->|over| DENY

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class REQ,PASS,DENY service;
    class G,T,B compute;

Budgets need post-call reconciliation because real token counts only land after the upstream finishes. The gateway records an estimated cost up front and adjusts when the response completes, occasionally letting a request exceed budget by a small slack rather than holding the client.

Sample code: adapter dispatch#

class GatewayHandler:
    def __init__(self, router, adapters, cache, vault, quota, redactor, events):
        self.router = router
        self.adapters = adapters
        self.cache = cache
        self.vault = vault
        self.quota = quota
        self.redactor = redactor
        self.events = events

    async def chat(self, tenant_id, api_key_id, req):
        await self.quota.consume(tenant_id, estimated_tokens=req.estimate_in())
        cache_key = self.cache.key(req)
        if cached := await self.cache.get_exact(cache_key):
            return self._emit(cached, hit="exact")
        if cached := await self.cache.get_semantic(req, threshold=0.97):
            return self._emit(cached, hit="semantic")

        try_list = self.router.plan(tenant_id, req)
        last_err = None
        for target in try_list:
            adapter = self.adapters[target.provider]
            creds = await self.vault.fetch(tenant_id, target.provider)
            try:
                async with adapter.stream(creds, target.model, req) as resp:
                    chunks = []
                    async for chunk in resp:
                        yield chunk
                        chunks.append(chunk)
                final = adapter.assemble(chunks)
                await self.cache.put(cache_key, final, ttl_s=3600)
                self.events.emit_usage(tenant_id, target, final)
                return
            except Exception as e:
                last_err = e
                self.events.emit_failover(tenant_id, target, e)
                continue
        raise last_err

Failure modes and scaling#

Failure Effect Mitigation
Single provider region down Spike in 5xx Health board down-weights, failover chain switches
Provider rate-limits a tenant 429 from upstream Token bucket throttles at gateway, optional spill to secondary provider
Vault unavailable Cannot fetch creds Short-lived cached lease (60 s) in encrypted memory, fail closed after lease expires
Cache poisoning Returns wrong answer TTL on entries, version cache key by prompt template, exclude tool-using turns
Runaway streaming Cost spike with no client Cancel upstream when client disconnects, hard token cap
Hot tenant Gateway pod CPU spike Per-tenant connection limit, autoscale on CPU + RPS

Scale-out is horizontal. Gateway pods are stateless and behind anycast; all state lives in Redis (quotas, caches) and Postgres (config, policies). The hot path is Redis: token bucket checks must be sub-millisecond, so quota lives in a Redis cluster with read replicas in each region. Provider adapters are async generators, so a single pod can hold tens of thousands of in-flight streaming connections on modest hardware.

Cost considerations#

The gateway business case is largely about cost recovery for customers. The gateway itself costs little; what it saves matters.

  • Gateway compute: 50k req/s peak, ~100 mid-size pods, ~$10k/month.
  • Redis cluster: 100 GB hot data, ~$3k/month managed.
  • Vector cache: 200 GB, ~$2k/month.
  • ClickHouse for logs: 10 TB/month ingest, ~$5k/month.
  • Total gateway opex: ~$20k/month for 50k req/s.

Customer savings come from caching and routing. A 30% semantic cache hit rate on chat workloads saves the customer roughly 30% of LLM spend; for a customer spending $100k/month on Anthropic, that is $30k/month saved at a fraction of that price in gateway fees. Cost-aware routing to a cheaper model when the policy allows (say, Haiku vs Sonnet on simple tasks) can compound that 2x.

Latency budget#

Target: p99 gateway overhead under 50 ms (excluding upstream provider RTT).

Step Budget Notes
TLS termination + parse 2 ms HTTP/2, keep-alive
Auth (cached lookup) 1 ms Redis hit on hashed key
Quota check 2 ms Lua script in Redis
Exact cache lookup 3 ms Redis GET
Embed + semantic lookup 30 ms Only on exact miss
Vault fetch 2 ms Lease cached for 60 s
Adapter translate + send 5 ms JSON rewrite, persistent conn pool
Stream first byte from provider (upstream) Outside budget
Response post-process + log 5 ms Async log emit
Gateway overhead p99 ~50 ms Worst case includes embed

When the semantic cache misses, the first 30 ms of embed dominates; this is why simple workloads should disable semantic cache to stay under 10 ms p99.

Quick reference#

Functional requirements#

  • OpenAI-compatible API across multiple providers.
  • Streaming (SSE) end-to-end.
  • Routing: static, weighted, cost-aware, latency-aware, failover.
  • Per-tenant API keys, separate from provider keys.
  • Per-tenant rate limits and dollar budgets.
  • Semantic cache and exact-match cache.
  • Prompt template registry with versioning.
  • Full request/response logs with PII redaction.
  • Token usage analytics per tenant, model, route.

Non-functional#

  • Gateway overhead p99 under 50 ms.
  • 50k req/s peak.
  • 99.95% availability.
  • No provider keys in plaintext at rest.
  • Hot-reloadable routing config.

Capacity (back-of-envelope)#

  • 50k req/s peak, ~100 mid-size pods.
  • Redis cluster ~100 GB hot data.
  • Vector cache ~200 GB.
  • ClickHouse 10 TB/month ingest.
  • Gateway opex ~$20k/month at this scale.

Architecture pillars#

  • Edge: anycast LB, stateless gateway pods.
  • Auth + quota: hashed key lookup, Redis token bucket, daily budget ledger.
  • Caches: Redis exact, vector DB semantic, embedding service.
  • Router: policy engine, health board, cost model.
  • Adapters: per-provider translation, signing, streaming.
  • Vault: KMS-backed, short-lived leases, audited fetches.
  • Observability: usage events on Kafka, ClickHouse rollups.

Routing policies#

Strategy When to use
Static Single provider preference per tenant
Weighted A/B test or load balance across two providers
Cost-aware Prefer cheapest healthy that meets quality
Latency-aware Prefer lowest p95 in current minute
Failover chain Survive single-provider outage
Sticky session Multi-turn conversation cache warmth

Cache rules#

  • Exact: hash(model, normalized prompt, params). Hot path.
  • Semantic: embed + ANN, similarity threshold 0.95-0.97.
  • Skip cache for tool-using turns, high temperature, streaming-subscribed sessions.
  • TTL by content age and tenant policy; default 1 hour.

Security checklist#

  • Provider keys only in KMS-encrypted vault.
  • Per-request lease (or 60s memory cache, encrypted).
  • PII redaction in logs, not requests.
  • Customer-controlled logging toggle.
  • Audit log of all vault fetches.

Common pitfalls#

  • Holding provider keys in env vars; rotation is painful.
  • Caching tool-using responses; breaks downstream agents.
  • Ignoring upstream cancellation; client disconnects but LLM bill keeps running.
  • One adapter for all providers; tool schemas drift silently.
  • No sticky routing; prompt cache benefits evaporate.
  • Per-request quota check that is not atomic; race conditions over budget.
  • Logging full prompts without redaction; PII leak.

Refs#

  • Portkey docs, Helicone architecture posts, LiteLLM source code, OpenRouter pricing pages, Cloudflare AI Gateway docs.

FAQ#

What does an LLM gateway proxy do?#

It exposes a single OpenAI-compatible API in front of many providers, routing each request, enforcing rate limits and budgets, caching prompts, and recording token usage for billing.

How does a semantic cache work?#

Embed the prompt, search a vector store for similar past prompts within a cosine threshold, and return the cached completion. Saves cost on near-duplicate queries like FAQ traffic.

How do you handle provider failover?#

Define a priority list per request type; on 5xx, rate-limit, or timeout, retry the next provider. Sticky routing within a session keeps streaming responses coherent.

How are API keys protected?#

Store provider credentials in a KMS-backed vault and load decrypted only into the proxy process memory; tenants never see raw provider keys, only gateway-issued ones.

How do you achieve sub-50 ms gateway overhead?#

Cache key lookup in Redis, skip JSON re-parsing by streaming bytes, pin tenant routing config in-memory, and run the proxy close to providers to keep RTT under 5 ms.

How is per-tenant rate limiting enforced?#

A token bucket per (tenant, model) backed by Redis with Lua-script atomic decrement. Exceeded buckets return 429 immediately so providers see only legitimate traffic.

  • API Gateway: general gateway patterns the LLM gateway specializes
  • Rate Limiter: per-tenant token bucket and sliding window algorithms
  • Caching Strategies: exact and semantic caching tradeoffs for LLM responses