Design ChatGPT#

A design ChatGPT system design interview answer covers more than a thin wrapper over a model API. The real product keeps a tree of every message, streams tokens token-by-token over SSE, lets users regenerate or branch any assistant reply, summarizes old turns to fit the context window, runs moderation on both directions, and tracks token cost per user and per organization. OpenAI's ChatGPT serves around 200M weekly active users; Anthropic's Claude.ai, Google Gemini, and Microsoft Copilot solve the same problem with similar building blocks.

Problem statement (interviewer prompt)

Design a chat application like ChatGPT or Claude.ai. Users send messages, receive streaming responses from an LLM, can regenerate any assistant reply, branch the conversation, edit a prior user message, and resume the chat days later. Support 200M weekly active users, p95 first-token latency under 2 seconds, multi-turn history that survives the model context window, content moderation in both directions, and per-organization billing.

flowchart LR
  U([User<br/>browser, mobile])
  CDN[CDN + Anycast LB]
  GW[API Gateway<br/>auth, rate limit]
  CHAT[Conversation service]
  CONV[(Conversation tree<br/>sharded by user)]
  CTX[Context builder<br/>summarize, retrieve]
  MOD[Moderation<br/>in + out]
  LLM[LLM provider<br/>OpenAI, Anthropic, vLLM]
  SSE[SSE streamer]
  TITLE[Title job<br/>async]
  USAGE[(Usage ledger)]

  U --> CDN
  CDN --> GW
  GW --> CHAT
  CHAT --> CONV
  CHAT --> CTX
  CHAT --> MOD
  CHAT --> LLM
  LLM --> SSE
  SSE --> U
  CHAT -.async.-> TITLE
  CHAT -.events.-> USAGE

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class U client;
    class CDN,GW,SSE edge;
    class CHAT,CTX,MOD service;
    class TITLE compute;
    class CONV,USAGE datastore;
    class LLM external;

This is not a stateless API. Every assistant reply is a function of the entire prior conversation. The client never holds the canonical history; the server does, sharded by user. A request is just an append: a new user message, the active branch's tip, and the model parameters. The backend reconstructs the prompt, calls the model, streams the answer back, and persists both turns on the way out.

Streaming is the user-perceived feature. A 4 second non-streaming response feels broken; the same 4 seconds with the first token arriving in 600 ms feels instant. The chat service must hold an open SSE connection per active user, forward upstream tokens as they arrive, handle client disconnects without orphaning an LLM call, and propagate the final token usage for billing.

Regen and branching are the data model lever. A chat is not a linear array of messages, it is a tree. Each user turn has one or more assistant children; each edit of a user turn forks a new subtree under the same parent. Storing the tree explicitly makes regenerate, edit, and branch navigation trivial; storing a flat list and trying to bolt branches on later is the classic refactor.

Requirements#

Functional

Send a user message and receive a streaming assistant reply.
Multi-turn conversation history persisted server-side.
Regenerate any assistant reply, producing a sibling under the same parent.
Edit any user message, forking a new branch under its parent.
Navigate between branches with arrow controls.
Auto-generated conversation title after the first exchange.
Conversation list with title, last activity, model used, and pinned status.
Abort a streaming response cleanly without burning the full completion.
Pre-request input moderation and per-chunk output moderation.
Per-user and per-org quotas: requests/min, tokens/min, monthly budget.
Per-conversation model selection (GPT, Claude, Gemini, internal).

Non-functional

200M weekly active users; ~30M concurrent peak.
p50 time-to-first-token under 1 s, p95 under 2 s.
p50 end-to-end response under 8 s, p95 under 20 s.
99.9% availability for the chat path.
Per-user data isolation: no cross-user leakage in any shared cache.
Soft cap on per-message tokens (model-specific), hard cap on per-day spend.
90-day conversation retention by default, exportable.

Top-level architecture#

flowchart TB
  subgraph Client[Clients]
    WEB([Web app])
    MOB([Mobile app])
    SDK([Public API])
  end

  subgraph Edge[Edge tier]
    CDN[CDN + static assets]
    LB[Anycast LB]
    GW[API gateway<br/>auth, rate limit]
    SSE[SSE streamer<br/>sticky-ish]
  end

  subgraph Core[Core services]
    CHAT[Conversation service<br/>append + read]
    CTX[Context builder<br/>budget + summarize + retrieve]
    MODIN[Input moderation]
    MODOUT[Output moderation]
    ROUTER[Model router<br/>cost + capacity aware]
  end

  subgraph Async[Async workers]
    TITLE[Title generator]
    SUM[Summarizer<br/>rolling memory]
    EMBED[Embedding indexer<br/>per-conversation]
    USAGE[Usage roll-up]
  end

  subgraph Data[Data plane]
    CONV[(Conversation DB<br/>sharded by user_id)]
    MSG[(Message tree<br/>same shard)]
    SUMSTORE[(Summary store)]
    EMBDB[(Conversation embeddings<br/>vector store)]
    LEDGER[(Usage ledger<br/>ClickHouse)]
    EVENTS[[Kafka]]
  end

  subgraph LLM[Model tier]
    OAI[OpenAI fleet]
    ANT[Anthropic fleet]
    GEM[Gemini fleet]
    SELF[Self-hosted vLLM]
  end

  WEB --> CDN
  MOB --> CDN
  SDK --> LB
  CDN --> LB
  LB --> GW
  GW --> CHAT
  CHAT --> CTX
  CHAT --> MODIN
  CHAT --> ROUTER
  CTX --> MSG
  CTX --> SUMSTORE
  CTX --> EMBDB
  ROUTER --> OAI
  ROUTER --> ANT
  ROUTER --> GEM
  ROUTER --> SELF
  ROUTER --> MODOUT
  MODOUT --> SSE
  SSE --> WEB
  SSE --> MOB
  CHAT --> CONV
  CHAT --> MSG
  CHAT -.events.-> EVENTS
  EVENTS --> TITLE
  EVENTS --> SUM
  EVENTS --> EMBED
  EVENTS --> USAGE
  USAGE --> LEDGER
  EMBED --> EMBDB
  SUM --> SUMSTORE
  TITLE --> CONV

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class WEB,MOB,SDK client;
    class CDN,LB,GW,SSE edge;
    class CHAT,CTX,MODIN,MODOUT,ROUTER service;
    class TITLE,SUM,EMBED,USAGE compute;
    class CONV,MSG,LEDGER,SUMSTORE datastore;
    class EMBDB cache;
    class EVENTS queue;
    class OAI,ANT,GEM,SELF external;

Conversation storage model#

The single most important schema decision is to store messages as a tree, not a list. Every assistant reply is a child of a user message; regenerate creates a sibling under the same parent; editing a user turn forks a new subtree from the parent of the edited message.

flowchart TB
  ROOT[System prompt<br/>conversation root]
  U1[User turn 1<br/>What is mitosis?]
  A1[Assistant 1<br/>original answer]
  A1b[Assistant 1b<br/>regenerated answer]
  U2[User turn 2 on branch A1<br/>Explain meiosis next]
  U2alt[User turn 2 edited<br/>And what about meiosis?]
  A2[Assistant 2<br/>follow-up]
  A2alt[Assistant 2 on edit]

  ROOT --> U1
  U1 --> A1
  U1 --> A1b
  A1 --> U2
  A1 --> U2alt
  U2 --> A2
  U2alt --> A2alt

    classDef root fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef user fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef asst fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef alt fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    class ROOT root;
    class U1,U2 user;
    class A1,A2 asst;
    class A1b,U2alt,A2alt alt;

The active branch is the path from root to the currently selected leaf. The conversation row stores active_leaf_id; flipping branches is a write to that single column. Future context is always reconstructed by walking parent pointers from the active leaf up to root, then reversing.

conversations(
  id uuid PK,
  user_id uuid,             -- shard key
  org_id uuid,
  title text,
  active_leaf_id uuid,      -- current branch tip
  default_model text,
  system_prompt text,       -- per-conversation, optional
  created_at timestamptz,
  updated_at timestamptz,
  pinned bool,
  archived bool
);

messages(
  id uuid,
  conversation_id uuid,
  parent_id uuid,           -- null only for root
  user_id uuid,             -- shard key, same as conversation
  role text,                -- system | user | assistant | tool
  content jsonb,            -- text, attachments, tool calls
  model text,               -- model that produced it (assistant only)
  tokens_in int,
  tokens_out int,
  finish_reason text,       -- stop | length | content_filter | abort
  moderation jsonb,
  created_at timestamptz,
  PRIMARY KEY (user_id, conversation_id, id)
);

attachments(
  id uuid PK,
  message_id uuid,
  user_id uuid,             -- shard key
  kind text,                -- image | file | code
  blob_url text,
  bytes bigint,
  mime text
);

The sharding key is user_id, not conversation_id, because the hot read pattern is the conversation list view (all conversations for a user) and 99% of writes are to a single user's data. A conversation is always co-located with its owner. A user's typical conversation set fits in tens of megabytes, so a single shard handles millions of users.

Soft deletes use an archived flag plus a TTL job that hard-deletes after 30 days. A real delete removes the conversation row, message rows, attachment rows, and the per-conversation embedding index in one transaction; the usage ledger is keyed by event ID and is preserved for billing.

Context window management#

A model call gets a fixed budget of tokens, usually 8k to 200k depending on model. A long-running conversation will eventually exceed it. The context builder is the component that decides what to include in each call.

flowchart TB
  REQ[New user message]
  WALK[Walk active branch<br/>root to leaf]
  COUNT[Count tokens<br/>using tokenizer]
  CHECK{Fits in budget?}
  KEEP[Use full history]
  TRUNC[Trim oldest turns]
  SUM[Fetch rolling summary<br/>prepend as one msg]
  RAG[Embed query, retrieve<br/>top-K old turns]
  BUILD[Build final prompt]
  CALL[LLM call]

  REQ --> WALK
  WALK --> COUNT
  COUNT --> CHECK
  CHECK -- yes --> KEEP
  KEEP --> BUILD
  CHECK -- no --> SUM
  SUM --> TRUNC
  TRUNC --> RAG
  RAG --> BUILD
  BUILD --> CALL

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class REQ,BUILD service;
    class WALK,COUNT,CHECK,KEEP,TRUNC,SUM,RAG compute;
    class CALL external;

The standard strategy stack, in priority order:

System prompt and pinned messages are always included.
Most recent K turns are kept verbatim; freshness matters more than depth for chat coherence.
Rolling summary of older turns is generated by an async worker after each N messages and stored in summaries(conversation_id, summary_text, covers_through_message_id). It is inserted as a single synthetic system-role message at the top of the prompt.
Retrieval over the per-conversation embedding index pulls in older turns that match the current user message by cosine similarity. This is the chat-app version of RAG: the corpus is the conversation itself.
Hard truncation drops the oldest remaining turns until the budget fits, with a 10% safety margin for the response.

The budget is model_window - max_response_tokens - safety_margin. A 128k model with 4k reserved for response and 2k safety gives 122k tokens of input space. Trying to use 100% of that triggers length finish reasons and broken streams.

Context window management goes deeper on the tradeoffs of each strategy and the cost of summarizing too aggressively.

Streaming responses#

The single biggest UX lever is streaming the first token within 1 second. There are three viable protocols, and chat apps overwhelmingly pick SSE.

sequenceDiagram
  participant C as Client
  participant GW as Gateway
  participant CH as Chat service
  participant LLM as LLM provider

  C->>GW: POST /chat/completions {message}
  GW->>CH: forward
  CH->>CH: persist user msg, build context
  CH->>LLM: streaming completion
  LLM-->>CH: chunk 1 (token)
  CH-->>C: SSE data: {delta: "Hi"}
  LLM-->>CH: chunk 2
  CH-->>C: SSE data: {delta: " there"}
  LLM-->>CH: ... more chunks ...
  CH-->>C: SSE data: {delta: "!"}
  LLM-->>CH: stop
  CH->>CH: persist assistant msg, emit usage event
  CH-->>C: SSE data: [DONE]
  CH-->>C: close

SSE vs WebSocket vs chunked HTTP#

Protocol	Direction	Reconnect	Proxy friendly	Used by
SSE (text/event-stream)	server to client	built-in `Last-Event-ID`	yes, plain HTTP	ChatGPT, Claude.ai, Gemini
WebSocket	bidirectional	manual	needs upgrade support	some Copilot variants, voice modes
Chunked HTTP (gRPC-Web stream)	server to client	manual	gRPC needs translation	internal LLM gateways

SSE wins for chat because the only thing the server needs to push is tokens; the user message is already a POST. SSE works through CDNs, corporate proxies, and HTTP/2 multiplexing without special config. WebSocket adds bidirectional power that chat does not need, and the persistent socket-per-user cost is non-trivial at 30M concurrent users. Voice modes and tool-use streams with interruption do use WebSocket because the client also pushes audio frames or tool results.

Abort and backpressure#

The client must be able to stop a running response. The standard pattern is a separate DELETE /chat/sessions/{id} or the client closing the SSE connection. The chat service detects disconnect, cancels the upstream LLM call, persists the partial assistant message with finish_reason='abort', and emits the usage event for the tokens already generated.

Backpressure happens when the client cannot consume tokens as fast as the LLM produces them (slow phone network, browser tab in background). The streamer maintains a small in-memory buffer per connection; if it fills, the upstream is paused via a flow-control read pause. In practice browsers consume tokens at 100 KB/s easily, so this only matters for tens-of-thousands-of-tokens-per-second self-hosted models on slow networks.

SSE handler shape#

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import json

app = FastAPI()

async def stream_chat(req, conv_id, user_msg):
    user_id = req.state.user_id
    await persist_user_message(user_id, conv_id, user_msg)
    context = await build_context(user_id, conv_id)
    if not await moderate_input(user_msg):
        yield sse_event({"error": "input_filtered"})
        return

    assistant_buf = []
    in_tokens = count_tokens(context)
    try:
        async with llm.stream(model="gpt-4o", messages=context) as resp:
            async for chunk in resp:
                if await req.is_disconnected():
                    await resp.cancel()
                    break
                delta = chunk.delta
                if not await moderate_chunk(delta):
                    yield sse_event({"error": "output_filtered"})
                    return
                assistant_buf.append(delta)
                yield sse_event({"delta": delta})
        final = "".join(assistant_buf)
        out_tokens = count_tokens(final)
        await persist_assistant_message(
            user_id, conv_id, final,
            tokens_in=in_tokens, tokens_out=out_tokens,
            finish_reason=resp.finish_reason,
        )
        await emit_usage(user_id, in_tokens, out_tokens)
        yield sse_event({"done": True})
    except Exception as e:
        yield sse_event({"error": str(e)})

def sse_event(data):
    return f"data: {json.dumps(data)}\n\n"

@app.post("/v1/conversations/{conv_id}/messages")
async def post_message(conv_id: str, req: Request):
    body = await req.json()
    return StreamingResponse(
        stream_chat(req, conv_id, body["content"]),
        media_type="text/event-stream",
    )

Regenerate and branching#

Regen is a POST against the parent of an existing assistant message. The API accepts parent_id, generates a new completion, and creates a new message row whose parent_id equals the existing assistant's parent. The conversation's active_leaf_id updates to the new message.

sequenceDiagram
  participant C as Client
  participant CH as Chat service
  participant DB as Message tree
  participant LLM as LLM

  C->>CH: POST /regen {parent_id: u1}
  CH->>DB: read children of u1
  DB-->>CH: [a1]
  CH->>DB: insert a1b with parent_id=u1
  CH->>LLM: stream new completion
  LLM-->>CH: tokens
  CH-->>C: SSE stream of a1b
  CH->>DB: finalize a1b content
  CH->>DB: update conversation.active_leaf_id = a1b
  CH-->>C: done

The regen API shape:

// POST /v1/conversations/{conv_id}/regenerate
interface RegenerateRequest {
  parent_id: string;          // the user message to re-answer
  model?: string;             // optional override
  temperature?: number;
  system_prompt?: string;     // optional override
}

interface RegenerateResponse {
  message_id: string;         // new sibling under parent_id
  branch_index: number;       // e.g. 2 of 3
  branch_count: number;
  active_leaf_id: string;
}

// PATCH /v1/conversations/{conv_id}/branch
interface SetActiveBranchRequest {
  leaf_id: string;            // any descendant of conversation root
}

The UI shows arrow controls beside each assistant turn that has siblings: < 2/3 >. Clicking the arrow PATCHes the active branch. The client never needs the full tree; it requests the path from root to active leaf, which is a single recursive SQL CTE.

Edit of a user message is the same primitive: insert a new user message with the same parent_id as the edited message, then immediately generate an assistant reply under it. The original user message and its assistant descendants stay in the tree as an inactive branch.

Title and summary generation#

Two async jobs are triggered on conversation events.

flowchart LR
  EVT[message_appended event]
  K[[Kafka topic]]
  TITLE[Title generator]
  SUM[Summarizer]
  EMBED[Embedder]
  DB[(Conversation DB)]
  SUMDB[(Summary store)]
  VEC[(Vector index)]

  EVT --> K
  K --> TITLE
  K --> SUM
  K --> EMBED
  TITLE --> DB
  SUM --> SUMDB
  EMBED --> VEC

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    class EVT,EMBED service;
    class TITLE,SUM compute;
    class K queue;
    class DB,SUMDB datastore;
    class VEC cache;

Title job. Fires only on the first assistant completion. Sends the first user message and assistant response to a cheap model (e.g. GPT-4o mini or Haiku) with prompt = "Generate a 4-6 word title for this conversation. Output only the title.". The result is written to conversations.title and pushed to the open SSE channel as a title event so the sidebar updates without a refresh. This is a model-as-judge variant: a cheap model decides what is salient.

Summary job. Fires after every N=20 new messages, or when the total token count crosses a watermark. The worker reads the messages not yet summarized, appends to the rolling summary with prompt "Update this summary to include the new exchanges. Keep it under 800 tokens.", and writes back. The summary is what the context builder prepends when the full history overflows.

Embedding indexer. Per assistant turn, embeds the user message and assistant content separately and writes vectors to a per-conversation vector index. The context builder uses this for retrieval-augmented context when summarization alone is insufficient.

All three jobs are idempotent: replaying a Kafka event re-runs them harmlessly because they write by (conversation_id, message_id) and skip if the watermark has already advanced.

Rate limiting and quota#

Three tiers stack, each cheaper to evaluate than the next:

flowchart LR
  REQ[Request]
  G[Global cap<br/>per-region]
  U[Per-user limit<br/>req/min, tokens/min]
  O[Per-org budget<br/>monthly $]
  M[Per-model quota<br/>GPT-4 capacity]
  PASS[Allow]
  DENY[429 / 402]

  REQ --> G
  G -- under --> U
  G -- over --> DENY
  U -- under --> O
  U -- over --> DENY
  O -- under --> M
  O -- over --> DENY
  M -- under --> PASS
  M -- over --> DENY

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class REQ,PASS,DENY service;
    class G,U,O,M compute;

Global cap is a Redis token bucket per region, defending the cluster from runaway traffic.
Per-user limit uses a Lua-scripted sliding-window in Redis: {user_id}:rpm and {user_id}:tpm. The tpm bucket is debited by an estimate up front and reconciled with actual usage after the stream finishes; over-budget is allowed by a small slack to avoid mid-stream aborts.
Per-org budget reads the daily roll-up in the usage ledger. A request that would push the org over its monthly $ ceiling is rejected with 402 Payment Required and surfaces a banner to the admin.
Per-model quota is the hardest one: a given model fleet has finite capacity. GPT-4 may have a global tokens_per_minute ceiling that the chat service must respect. The router maintains a moving window of upstream 429s and shifts traffic to a fallback model when the primary saturates.

Rate limiter covers the token-bucket and sliding-window algorithms in depth.

Content moderation#

Moderation is two passes, one before the LLM and one during the stream.

flowchart LR
  IN[User message]
  MI[Input classifier<br/>fast model]
  GATE1{Safe?}
  REFUSE[Return refusal]
  CALL[LLM call]
  CHUNK[Token chunk]
  MO[Output classifier<br/>sliding buffer]
  GATE2{Safe?}
  STOP[Stop stream<br/>safe completion]
  SEND[Forward to client]

  IN --> MI
  MI --> GATE1
  GATE1 -- no --> REFUSE
  GATE1 -- yes --> CALL
  CALL --> CHUNK
  CHUNK --> MO
  MO --> GATE2
  GATE2 -- no --> STOP
  GATE2 -- yes --> SEND

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class IN,REFUSE,SEND,STOP service;
    class MI,GATE1,MO,GATE2 compute;
    class CALL external;

Input moderation runs on the user message before the LLM is touched. A fast classifier (OpenAI moderation endpoint, internal small model) returns category scores; messages crossing a threshold are blocked with a refusal. This saves the LLM cost and prevents prompt-injection content from reaching the model. Prompt injection defense covers the deeper layered defenses for tool-using agents.

Output moderation runs on a sliding 100-token buffer of the assistant's stream. The classifier is called every few chunks; on a flag, the stream is replaced with a canned safe completion mid-token. Replacing rather than truncating prevents the user from seeing partial unsafe content. The full transcript is logged for review.

Severe violations (CSAM, terrorism uplift) feed a moderation case queue. A user accumulating cases over a threshold is rate-limited, then suspended, then escalated to human review with the conversation evidence.

Cost and observability#

A token-counted system is the only one you can reason about.

flowchart LR
  CHAT[Chat service]
  TOK[Tokenizer<br/>tiktoken / claude-tok]
  EV[Usage event<br/>tenant, model, in/out]
  K[[Kafka]]
  RAW[(ClickHouse raw)]
  ROLL[Roll-up worker]
  LED[(Usage ledger<br/>per-user, per-org)]
  DASH[Dashboards<br/>billing, admin]
  ALERT[Cost alerts]

  CHAT --> TOK
  CHAT --> EV
  EV --> K
  K --> RAW
  RAW --> ROLL
  ROLL --> LED
  LED --> DASH
  LED --> ALERT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class CHAT,TOK service;
    class EV,ROLL compute;
    class K queue;
    class RAW,LED datastore;
    class DASH,ALERT obs;

Tokens are counted with the model's own tokenizer (tiktoken for OpenAI, the Anthropic tokenizer for Claude). Counting on the gateway is required because providers' returned token counts arrive only at the end of the stream; the system needs an early estimate for quota debiting. Reconciliation runs at the end: if the actual exceeds the estimate, the ledger is adjusted up; if it falls short, the slack is credited back.

Roll-ups produce per-(user, hour), per-(org, day), and per-(model, hour) buckets. Dashboards surface: cost per active user, cost per conversation, cost per model, and the long tail of expensive conversations (a few users with huge contexts can dominate spend).

Cost-aware model routing is the cheapest optimization. A 4-word user message asking "what is the date" does not need a frontier model; route it to a small model and save 95%. A simple classifier or a routing prompt picks the cheapest model that meets quality; only complex turns go to the frontier tier. This single change typically cuts marginal cost by 30 to 60% for general-purpose chat workloads.

LLM gateway proxy is the component that owns the multi-provider routing, semantic caching, and key management once you have more than two upstream models.

Multi-tenant safety#

A chat platform is a giant pile of private conversations. The default for any shared resource must be isolation by user.

Caches keyed by user ID. Any cache key that omits user_id is a bug. A semantic cache that returns a different user's stored answer because the prompt is similar is a privacy incident, full stop. For chat, the safest stance is to disable cross-user semantic caching entirely and rely on per-conversation caching only (which is exact-match anyway).
Embedding indexes scoped per conversation. A vector store query must always filter by conversation_id. A global cross-user index for "memory" features needs explicit user opt-in and a per-user namespace.
System prompts not user-controllable in shared products. The base system prompt is server-controlled; user customizations are appended after, never above, so a user cannot impersonate the system role.
Prompt-injection defense at tool boundaries. When the assistant calls tools (browse the web, run code), tool output is treated as untrusted and never executed as new instructions. See tool use and function calling and the deeper prompt injection defense page.
Audit log of every assistant action that touches external state. Tool calls, file writes, and external API calls are logged with the message ID that triggered them, the user ID, and the response.

Real-world numbers#

For interview calibration, the headline numbers are:

ChatGPT weekly active users: ~200M (as of late 2025).
Concurrent peak: ~30M.
Average messages per session: 8 to 15.
p50 time-to-first-token: under 1 s for cached / small models, 1 to 2 s for frontier.
Average assistant response: 300 to 800 tokens.
Median conversation length: 6 turns; long tail to thousands of turns.
Daily LLM compute spend at this scale: estimated mid hundreds of millions of dollars per year across all providers in the chat-app category combined.

These are useful for back-of-envelope work. A senior IC answer should land within an order of magnitude on each.

Capacity estimation#

200M WAU, 5 sessions/week each, 10 messages/session = 10B user messages/week, ~16k messages/second average, ~50k peak.
Assistant token output averages 500 tokens/turn = 8M tokens/second of generation peak.
Storage: 10B messages/week, ~1 KB each = 10 TB/week of conversation data, ~500 TB/year before compression and TTL.
Embedding storage: 1 vector (1536 dim, float16) per message = ~3 KB/message; full retention is ~30 TB/year.
Per-conversation embedding indexes are small; a 100-message conversation is 300 KB of vectors.
Network: 50k SSE streams concurrent at peak, average 2 KB/s/stream during generation = 800 Mbps egress. Plus image attachments and the long tail.

Failure modes#

Failure	Effect	Mitigation
Upstream LLM provider 5xx	Stream errors mid-response	Router fails over to next provider, with sticky-per-conversation reset on fail
Single shard hot user	DB write contention	Per-user write rate limit, async batching of non-critical writes (titles, usage)
SSE connection dropped	Half-streamed assistant message	Persist a partial assistant row with `finish_reason='abort'`; client can resume by reading active leaf
Context overflow	Model returns `length` finish	Context builder enforces budget; if hit, retry with a more aggressive summarize step
Moderation classifier slow	Adds latency to first token	Classifier runs in parallel with first LLM tokens, blocks the stream only if it flags
Title job stuck	Conversation shows "New chat" forever	Idempotent retry from Kafka with exponential backoff; fallback to client-side title from first user message
Cross-user leakage in cache	Privacy incident	Cache keys include `user_id`; semantic cache disabled across users; integration tests prevent regression
Bulk export storm	Read load on user shards	Export job reads from a read replica, throttled per user

Common interview tradeoffs#

Store the tree or a list? Tree wins because regen and branching are first-class. A list works for a v0 but is a painful refactor at scale.
SSE or WebSocket? SSE for chat; WebSocket only for voice / live-tool modes.
Summarize older turns or use a long-context model? Both. Summarization is cheap and works at any model size; long context is for the current task window. Pure long-context-only burns tokens and money for stale data.
Server-side history or client-side? Server-side. Client-side breaks resume on a new device, breaks multi-device sync, and breaks share-conversation.
One model per conversation or per turn? Per turn is more flexible (router can downshift on simple turns) but requires a stable per-conversation context format that does not bake in model-specific quirks.
Sync title generation or async? Async. Blocking the first response on title generation adds 500 ms for no UX value.

FAQ#

How does ChatGPT handle multi-turn conversation history?#

Every user turn and assistant response is stored as a node in a conversation tree keyed by message ID and parent ID. On each new request the backend walks the active branch, applies a context budget, and sends only the messages that fit in the model window.

Why does ChatGPT use SSE instead of WebSocket for streaming?#

SSE is one-way server-to-client over plain HTTP, which is enough for token streaming and easier to scale through CDNs and load balancers. WebSocket is bidirectional but expensive to keep alive at hundreds of millions of users, so chat apps use SSE for the response stream and a separate POST for the user message.

How does regenerate or branch a response work?#

Regenerate creates a new child message that shares the same parent as the existing assistant turn. The conversation tree now has two children under one parent, and the UI lets the user flip between branches with arrow controls. Only one branch is active at a time and feeds future context.

How does ChatGPT keep older conversations inside the context window?#

The conversation service applies a context budget per request. Messages older than the budget are either dropped, summarized into a short rolling summary message that lives at the top of the prompt, or replaced with a retrieval lookup over the conversation embedding index.

How is the conversation title generated?#

After the first user and assistant turn, an async job sends the pair to a cheap model with a prompt like generate a 4 word title. The result is written back to the conversation row and pushed to the client over the same SSE channel.

How does ChatGPT handle abuse and content moderation?#

Each user message runs through a fast moderation classifier before the LLM call, and each assistant chunk runs through an output filter as it streams. Flagged content is replaced with a safe completion or a refusal, and severe violations are logged for human review and rate limiting.

How are costs tracked per user and per organization?#

The chat service counts tokens for both prompt and completion using the model tokenizer, emits a usage event with tenant ID, model, and tokens to Kafka, and rolls it up in a usage ledger. Quotas and billing read from the ledger, and dashboards expose per-org spend and per-model breakdowns.

Agent Loop and ReAct: agent-style chat is a superset that adds tool calls and multi-step reasoning to the basic conversation loop
Context Window Management: truncation, summarization, and retrieval strategies for long conversations
Tool Use and Function Calling: how a chat app adds browsing, code execution, and tool turns to the message tree
LLM Gateway Proxy: the multi-provider router, semantic cache, and key vault behind a multi-model chat app
Prompt Injection Defense: layered defenses for tool-using chat and shared context safety

Quick reference#

Functional requirements#

Send message and receive streaming assistant reply.
Multi-turn history persisted as a tree, sharded by user.
Regenerate, edit, and branch with arrow navigation.
Auto-generated title, model selection per conversation.
Abort, retry, and resume after disconnect.
Pre-request input moderation, in-stream output moderation.
Per-user, per-org, per-model quotas.

Non-functional#

200M WAU, 30M concurrent peak.
p50 time-to-first-token under 1 s; p95 under 2 s.
p50 end-to-end under 8 s; p95 under 20 s.
99.9% availability for the chat path.
Per-user isolation; no shared semantic cache across users.

Storage shape#

conversations(id, user_id, title, active_leaf_id, default_model, ...).
messages(user_id, conversation_id, id, parent_id, role, content, model, tokens_in, tokens_out, finish_reason, ...).
Shard key: user_id.
Soft delete + 30 day TTL hard delete; usage ledger preserved.

Context strategies#

Strategy	When to use	Cost
Full history	Conversation fits in budget	Linear in tokens
Last K turns + summary	Conversation overflows window	One summary call per N messages
Retrieval over conversation embeddings	Old turns relevant to current question	One embedding + ANN query per request
Hard truncation (oldest first)	Last resort	Free but degrades coherence

Streaming protocols#

Protocol	Direction	Use it for
SSE	server to client	text token streaming, default chat
WebSocket	bidirectional	voice, live tool, interruptible audio
Chunked HTTP	server to client	server-to-server gRPC streams

Regen and branching API surface#

Endpoint	Purpose
`POST /v1/conversations`	Create conversation
`POST /v1/conversations/{id}/messages`	Append user message, stream reply
`POST /v1/conversations/{id}/regenerate`	Sibling assistant under same parent
`POST /v1/conversations/{id}/edit-user`	Sibling user message, then auto reply
`PATCH /v1/conversations/{id}/branch`	Set `active_leaf_id`
`GET /v1/conversations/{id}/tree`	Return tree or active path
`DELETE /v1/conversations/{id}/stream`	Abort active stream

Quotas#

Layer	Storage	Rejection code
Global per-region cap	Redis token bucket	503
Per-user rpm / tpm	Redis sliding window, Lua	429
Per-org monthly budget	Usage ledger	402
Per-model fleet capacity	Router moving window	429 + failover

Cost numbers (illustrative, late 2025)#

Model	Input $/1M	Output $/1M	Use for
GPT-4o	$2.50	$10	Default frontier
GPT-4o mini	$0.15	$0.60	Cheap routing tier
Claude Sonnet	$3	$15	Frontier alt
Claude Haiku	$0.25	$1.25	Title, classify, summarize
Self-hosted Llama 70B	~$0.30 / hour GPU	(compute)	High-volume internal

Real-world numbers to cite#

ChatGPT WAU ~200M; concurrent peak ~30M.
p50 first-token < 1 s; p95 < 2 s on frontier.
8 to 15 messages per session average.
Average assistant turn 300 to 800 tokens.

Common pitfalls#

Storing chat as a flat list; regen and branching become impossible.
Sending entire history to model every turn; cost explodes after 50 messages.
Caching responses across users in a shared semantic cache; privacy incident.
WebSocket for plain text streaming; needless socket cost at scale.
Title generation on the synchronous path; adds 500 ms for no value.
Forgetting to cancel the upstream LLM call on client disconnect; pays for unwatched tokens.
One model for everything; pay frontier prices for "hello" turns.
No output moderation; assistant streams unsafe content to user before review.
Per-user data in shared embedding namespace; cross-user leakage at retrieval.

Refs#

OpenAI ChatGPT engineering blog posts on streaming and conversation storage.
Anthropic claude.ai design notes; messages API and streaming docs.
Google Gemini Apps architecture talks.
LangChain, LlamaIndex, and Vercel AI SDK source code for client-side patterns.
tiktoken and anthropic-tokenizer libraries for token accounting.
HTML SSE spec (WHATWG) and HTTP/2 push behavior notes.