Streaming LLM Responses#

Q: How do you stream responses from an LLM?

Use Server-Sent Events over HTTP for unidirectional token streaming, WebSockets when the client needs to send abort or mid-stream messages, or chunked HTTP with fetch and ReadableStream. Each chunk carries a small delta of tokens that the client appends to a buffer.

Q: Why does streaming LLM output feel faster even when total latency is the same?

Perceived latency drops because the user starts reading at the time-to-first-token, typically 200 to 800 milliseconds, instead of waiting for the full completion. Reading speed is around 5 tokens per second so streaming masks generation time up to that rate.

Q: What is the difference between SSE and WebSocket for LLM streaming?

SSE is unidirectional over plain HTTP/1.1 with automatic reconnect and an EventSource browser API. WebSocket is full-duplex over a single upgraded TCP connection, letting the client send abort frames or follow-up messages without opening a new request.

Q: How do you cancel an in-flight LLM stream?

On SSE close the underlying response or call AbortController on fetch. On WebSocket send a cancel frame. The server detects the closed socket and stops generation to free the GPU. Detection takes a few hundred milliseconds and you still pay for tokens already emitted.

Q: Why does Cloudflare or nginx sometimes break LLM streaming?

Default proxy and CDN configs buffer chunked HTTP responses before forwarding to clients, so tokens arrive in one big lump instead of incrementally. Set Content-Type to text/event-stream and disable response buffering with X-Accel-Buffering no on nginx or Cloudflare's response buffering flag.

Q: How do you stream structured JSON from an LLM?

Use a streaming JSON parser that accepts partial input, like partial-json in TypeScript or json-stream in Python. The parser yields complete fields as soon as they close, so the client can render tool-call arguments or table rows incrementally before the full object is generated.

Streaming LLM responses is the production pattern that hides 5 to 30 seconds of generation latency behind a 200 millisecond time-to-first-token. The model emits tokens as it generates them, the transport ships each token to the client as a small event, and the UI types them out. Without streaming an LLM chat product feels broken; with it, the same model feels real-time.

flowchart LR
  U([User]) --> APP[App backend]
  APP --> LLM[LLM inference]
  LLM -.tokens.-> APP
  APP -.SSE / WS / chunks.-> U

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class U client;
  class APP service;
  class LLM compute;

Three transports dominate: Server-Sent Events for the 90% case, WebSockets where the client needs to talk back, and chunked HTTP with a ReadableStream for SDKs that wrap fetch. Each provider has its own delta format (OpenAI choices[].delta, Anthropic event: content_block_delta, Vercel AI SDK 0:"text") and the client must know which to parse. On top of the transport, production code has to handle abort, retry, backpressure on the GPU, CDN buffering, sticky multi-region routing, partial JSON for tool calls, and a typed-out UI that still scrolls and copies cleanly.

Problem statement

A chatbot product calls a model that takes 8 seconds to produce a 400-token answer. Users abandon at 3 seconds of blank screen. The team wants the user to see the first word in under 500ms, support a cancel button, work behind Cloudflare and nginx, stream tool calls for an inline UI, and survive WiFi drops without dropping the whole conversation. Design the streaming LLM responses stack end to end.

Why streaming matters: TTFT vs total time#

Two latency numbers describe a streaming LLM call:

Time-to-first-token (TTFT): the gap between sending the request and the first token arriving. Typical: 200 to 800 milliseconds depending on model size, batch position, and whether the prompt cache hits.
Total completion time: TTFT plus output tokens divided by inter-token rate. Typical inter-token delay is 10 to 50 milliseconds, so a 400-token answer takes another 4 to 20 seconds.

Reading speed for adults sits around 4 to 6 tokens per second of displayed text. If the model emits at 30 tokens per second, the user is reading slower than the model is generating, and total perceived latency is TTFT plus reading time, not TTFT plus generation time. That is the entire reason streaming exists. The same 8-second job feels like a 400-millisecond job because the user is busy reading the first sentence while the model writes the next four.

Metric	Non-streaming	Streaming
Blank screen	8s	0.4s
Abandon rate at 3s	30%	<1%
Perceived latency	8s	0.4s + reader-paced
Server cost	identical	identical

The server cost is identical because streaming does not change generation work; it only changes when bytes go on the wire.

The three transports#

flowchart TB
  subgraph SSE[Server-Sent Events]
    S1[HTTP/1.1 long response<br/>Content-Type: text/event-stream]
    S2[Unidirectional server to client]
    S3[Built-in EventSource API<br/>auto-reconnect]
  end
  subgraph WS[WebSocket]
    W1[HTTP upgrade then frames]
    W2[Full-duplex]
    W3[Client sends abort or follow-up]
  end
  subgraph CH[Chunked HTTP + ReadableStream]
    C1[fetch with Transfer-Encoding: chunked]
    C2[Reader.read in browser]
    C3[Used by OpenAI / Anthropic SDKs]
  end

  classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class S1,S2,S3 edge;
  class W1,W2,W3 service;
  class C1,C2,C3 compute;

Server-Sent Events (SSE)#

SSE is the default for OpenAI, Anthropic, Cohere, and almost every modern LLM API. The server sets Content-Type: text/event-stream, writes one event per line block, and never closes until done. The browser's EventSource automatically reconnects on disconnect using the Last-Event-ID header.

event: message_start
data: {"type":"message_start","message":{"id":"msg_01","model":"claude-opus-4-7"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" world"}}

event: message_stop
data: {"type":"message_stop"}

Pros: works through every HTTP proxy that respects Cache-Control: no-cache, no upgrade, easy to debug with curl. Cons: unidirectional, 6 concurrent connections per origin in HTTP/1.1 browsers (HTTP/2 multiplexes), and the EventSource API does not send custom headers (workaround: use fetch + ReadableStream and parse SSE manually).

WebSocket#

A full-duplex socket built on a TCP connection promoted from HTTP via Upgrade: websocket. The client can send a cancel frame, additional messages, or out-of-band signals while the server is mid-stream.

Pros: bidirectional means a user can hit cancel and the abort frame travels back over the same socket the tokens are streaming on. No 6-connection cap, since one socket carries everything. Cons: not transparently cached by HTTP infra, needs sticky session affinity through every proxy, harder to debug, and middleboxes sometimes drop idle WebSocket connections at 60 to 120 seconds.

Used by ChatGPT's web app for the cancel button, by Claude.ai for typing notifications, and by anything that needs the client to send messages mid-completion.

Chunked HTTP + ReadableStream#

Modern browsers expose fetch().body.getReader(), which returns chunks of the response body as they arrive. Underneath, the server uses Transfer-Encoding: chunked over HTTP/1.1 or DATA frames over HTTP/2. There is no special event framing: the client parses whatever bytes arrive (often the same SSE format, sometimes a custom NDJSON or Vercel AI SDK protocol).

The OpenAI Node SDK uses this pattern. The Vercel AI SDK does too, with its own line-prefixed format: 0:"hello" for text deltas, 1:{...} for tool calls, 2:{...} for errors.

Pros: native to fetch, supports custom headers (unlike EventSource), works on HTTP/2 and HTTP/3 without changes. Cons: no automatic reconnect, no built-in event parser, must roll your own.

Provider format conventions#

Provider	Transport	Delta shape
OpenAI Chat Completions	SSE	`{"choices":[{"delta":{"content":"tok"}}]}`
OpenAI Responses API	SSE	`event: response.output_text.delta` with `{"delta":"tok"}`
Anthropic Messages	SSE	`event: content_block_delta`, `data: {"delta":{"text":"tok"}}`
Google Gemini	SSE	`data: {"candidates":[{"content":{"parts":[{"text":"tok"}]}}]}`
Vercel AI SDK	chunked HTTP	`0:"tok"` per line, `9:{...}` for tool call, `d:{...}` finish
Cohere	SSE	`{"event_type":"text-generation","text":"tok"}`

Three shape patterns dominate: delta-merge (OpenAI Chat), typed events (OpenAI Responses, Anthropic, Vercel), and full-snapshot-on-every-chunk (Gemini, in older formats). The merge shape sends only the change; the snapshot shape resends the entire message each chunk, which makes parsing simpler but bandwidth higher.

SSE sequence in detail#

sequenceDiagram
  autonumber
  participant C as Browser
  participant LB as Load balancer / CDN
  participant S as App server
  participant L as LLM provider

  C->>LB: GET /chat/stream (Accept: text/event-stream)
  LB->>S: forward, sticky to one backend
  S->>L: POST messages stream=true
  L-->>S: event: message_start
  S-->>C: data: {"type":"message_start"}
  loop every token
    L-->>S: event: content_block_delta
    S-->>C: data: {"delta":{"text":"tok"}}
  end
  C->>C: append to UI buffer, render
  L-->>S: event: message_stop
  S-->>C: data: {"type":"done"}
  S-->>C: close response

Two streams stack: provider-to-app and app-to-client. The app server is mostly a pipe but it must:

Authenticate the user before opening the upstream call.
Inject the system prompt, conversation history, tools.
Add or rewrite event types so the client format is stable across providers.
Multiplex with reasoning, citations, or guardrail events.
Track token usage on the final event for billing.

Server code: Node SSE handler#

import express from "express";
import Anthropic from "@anthropic-ai/sdk";

const app = express();
const anthropic = new Anthropic();

app.post("/chat/stream", express.json(), async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache, no-transform");
  res.setHeader("Connection", "keep-alive");
  res.setHeader("X-Accel-Buffering", "no");
  res.flushHeaders();

  const aborted = new AbortController();
  req.on("close", () => aborted.abort());

  try {
    const stream = await anthropic.messages.stream({
      model: "claude-opus-4-7",
      max_tokens: 1024,
      messages: req.body.messages,
    }, { signal: aborted.signal });

    for await (const event of stream) {
      if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
        res.write(`event: token\ndata: ${JSON.stringify({ text: event.delta.text })}\n\n`);
      }
    }
    const final = await stream.finalMessage();
    res.write(`event: usage\ndata: ${JSON.stringify(final.usage)}\n\n`);
    res.write(`event: done\ndata: {}\n\n`);
  } catch (err) {
    if (err.name !== "AbortError") {
      res.write(`event: error\ndata: ${JSON.stringify({ message: err.message })}\n\n`);
    }
  } finally {
    res.end();
  }
});

Three things to notice. X-Accel-Buffering: no tells nginx not to buffer (Cloudflare needs a separate setting). The AbortController ties client-disconnect to upstream-cancel so a closed browser frees the GPU. The finalMessage() call yields token usage even after the stream ends because Anthropic emits a message_delta event with billing info before the stop.

Client code: EventSource and fetch readers#

// Option A: EventSource (no custom headers, auto-reconnect)
const es = new EventSource("/chat/stream?session=abc");
es.addEventListener("token", (e) => {
  const { text } = JSON.parse(e.data);
  ui.append(text);
});
es.addEventListener("done", () => es.close());
es.addEventListener("error", () => {
  // EventSource auto-retries; close manually if we want to stop
  if (es.readyState === EventSource.CLOSED) ui.showRetryButton();
});

// Option B: fetch + ReadableStream (custom headers, manual parse)
const ctrl = new AbortController();
const resp = await fetch("/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json", "Authorization": `Bearer ${token}` },
  body: JSON.stringify({ messages }),
  signal: ctrl.signal,
});

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  let idx;
  while ((idx = buffer.indexOf("\n\n")) >= 0) {
    const eventBlock = buffer.slice(0, idx);
    buffer = buffer.slice(idx + 2);
    const lines = eventBlock.split("\n");
    const event = lines.find(l => l.startsWith("event: "))?.slice(7);
    const data = lines.find(l => l.startsWith("data: "))?.slice(6);
    if (event === "token" && data) ui.append(JSON.parse(data).text);
  }
}

// abort button:
cancelButton.onclick = () => ctrl.abort();

The \n\n block boundary is the SSE convention. Partial UTF-8 sequences are why TextDecoder is created with stream: true: a multibyte glyph split across chunks would corrupt otherwise.

Backpressure: the GPU side of slow readers#

Backpressure happens when the client reads slower than the server emits. TCP applies it automatically: the kernel's send buffer fills, write() blocks, the app server stops draining the upstream LLM SDK, and the upstream SDK's queue fills. The provider then sees a slow consumer.

Why this matters for cost: an in-flight LLM generation pins KV cache GPU memory. Every token generated needs the running attention state for every prior token. A 4K-token context at FP16 is roughly 200 MB per request on a 70B model. If the client is on a 3G phone reading at 1 KB/s and the server emits at 10 KB/s, the model holds GPU memory for 10x longer than it would on a fast network, blocking other concurrent users.

Mitigations:

Hard server timeout on the upstream call: if no token movement for 30 seconds, abort.
Idle write timeout on the client socket: if write blocks for 5 seconds, close.
Token-rate cap: deliberately throttle inter-token writes so a slow client looks the same as a fast one (rare, but used by some prompt-cache aware proxies).
Drop the prefix cache, keep position: providers like Anthropic let you stream into a prefilled assistant turn so a reconnect resumes near where the disconnect happened, but only inside provider TTL windows.

Abort flow and detection latency#

sequenceDiagram
  autonumber
  participant C as Client
  participant S as App server
  participant L as LLM
  participant GPU as GPU worker
  C->>S: stream open
  S->>L: messages stream
  L->>GPU: generation in progress
  GPU-->>L: tok 1
  L-->>S: delta
  S-->>C: delta
  C--xS: socket close (cancel)
  S->>S: detect close on next write
  S->>L: abort signal
  L->>GPU: stop generation
  GPU-->>L: released
  Note over C,GPU: detection takes 100-500ms<br/>tokens emitted in window are billed

Three latencies stack:

Client to server: TCP RST or FIN reaches the server in one RTT, but the server only notices on its next write (the kernel does not push a notification). Time-to-detect: 0 to inter-token-delay, typically 10 to 100 ms.
Server to provider: the abort signal travels as a fresh request to the provider's cancel endpoint, or as AbortController.abort() on the SDK socket. Another RTT plus the provider's handler delay: 50 to 300 ms.
Provider to GPU: even after the API receives the abort, the in-flight token batch on the GPU has to finish before stopping. Batched serving means 1 to 5 more tokens get emitted (and billed).

Net detection latency: 100 to 500 ms, during which the user is billed for whatever tokens leak through. Anthropic and OpenAI both charge for emitted tokens up to the abort point, not the requested max_tokens. This is why showing a snappy cancel button is cheap; the actual cost savings come from real disconnects on long generations.

Retry and resume strategies#

A streaming response is not idempotent in the usual sense: the model is stateful inside one call, and most providers do not let you replay only the missing tail. Three patterns:

Full retry on early failure#

If the stream dies before the first token, treat it like a regular HTTP failure: exponential backoff, retry the whole request with the same prompt. This is safe because the user has not seen any output yet.

Best-effort partial completion#

If the stream dies mid-response, keep what you have, surface a "regenerate" button. Most chat UIs do this because resuming is fragile and rebilling is expensive.

Resumable streams (Anthropic Resume Events, experimental)#

Anthropic supports a Resume header pattern where you pass the last message_id and the server picks up from that token position, replaying any events the client missed. OpenAI does not support this in 2026. The pattern works because the provider keeps the generation state hot in its KV cache for a short TTL (typically 30 to 60 seconds).

// Pseudo-code for resume
let lastEventId = localStorage.getItem("lastEventId");
const es = new EventSource("/chat/stream", {
  headers: { "Last-Event-ID": lastEventId },
});
es.addEventListener("token", (e) => {
  lastEventId = e.lastEventId;
  localStorage.setItem("lastEventId", lastEventId);
  ui.append(JSON.parse(e.data).text);
});

EventSource sends Last-Event-ID automatically on reconnect. The app server uses it to skip already-emitted events.

JSON streaming for structured output#

Tool calls and structured outputs ship as JSON, which is not delimitable until the closing brace. But the model emits tokens that look like JSON characters one at a time. A partial-JSON parser turns this stream into a usable object.

sequenceDiagram
  autonumber
  participant LLM
  participant App
  participant UI
  LLM-->>App: {"
  LLM-->>App: name":"Bos
  LLM-->>App: ton","temp":7
  App->>App: parse-partial -> {name:"Boston",temp:7 (incomplete)}
  App-->>UI: render {name:"Boston"}
  LLM-->>App: 2}
  App->>App: parse-partial -> {name:"Boston",temp:72}
  App-->>UI: render full

Libraries: partial-json (TypeScript/Vercel AI SDK), json-stream-parser, streaming-json (Python). The parser treats truncated tokens as soft completions: an open string becomes the longest-prefix string, an open object becomes the partial set of completed fields.

import { parse } from "partial-json";

let buffer = "";
for await (const chunk of stream) {
  buffer += chunk.delta;
  try {
    const partial = parse(buffer);
    ui.updateForm(partial);  // safe to render
  } catch {
    // not enough yet, keep buffering
  }
}

OpenAI tool calls stream as delta.tool_calls[0].function.arguments += "tok" and Anthropic streams input_json_delta events. Both terminate on a content block stop, after which a strict parse is safe.

Use cases that benefit:

Form filling: each field appears in the UI as soon as its value closes.
Table rendering: rows appear one at a time during a findAll tool call.
Code generation: language fence opens immediately, syntax highlighting starts.
Reasoning traces: long chain-of-thought streams as a collapsible section.

CDN and proxy considerations#

flowchart TB
  C([Browser]) --> CF[Cloudflare]
  CF --> NG[nginx ingress]
  NG --> APP[App server]
  APP --> LLM[LLM provider]

  CF -->|default: buffer chunked| BAD[Tokens arrive in lump]
  CF -->|text/event-stream: stream| GOOD[Tokens arrive live]
  NG -->|proxy_buffering on: lump| BAD
  NG -->|proxy_buffering off: live| GOOD

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class C client;
  class CF,NG edge;
  class APP service;
  class LLM compute;
  class BAD datastore;
  class GOOD compute;

Default CDN and reverse-proxy behaviour is to buffer chunked HTTP responses to reduce origin pressure and improve compression. For streaming this is fatal: tokens batch up and arrive in one chunk after the full response completes.

Required settings:

Layer	Setting	Value
nginx	`proxy_buffering`	`off` for the streaming location
nginx	`proxy_request_buffering`	`off`
nginx	response header	`X-Accel-Buffering: no`
Cloudflare	"Response Buffering"	off (auto if `Content-Type: text/event-stream`)
Cloudflare	Workers `Response`	use `TransformStream` not `arrayBuffer()`
AWS ALB	`idle_timeout`	raise to 4000s for long generations
CloudFront	uses chunked passthrough	works for SSE with cache disabled
HTTP/1.1	`Connection: keep-alive`, `Transfer-Encoding: chunked`	required
HTTP/2	no special change; DATA frames stream natively	works

The text/event-stream content type is the signal most CDNs use to skip buffering. Setting it correctly is the single highest-leverage line in your code.

Compression interactions: gzip works fine for SSE because nginx and Cloudflare flush per chunk when Content-Type: text/event-stream. Brotli sometimes batches; test before relying on it.

Multi-region routing and stickiness#

A streaming response is a stateful TCP connection that lives for seconds to minutes. Every layer in front of the app must route the entire stream to the same backend.

DNS load balancing: use geo-routing to send the user to the nearest region; after that, the TCP connection stays put.
L7 LB session affinity: hash on IP, cookie, or session ID. Without affinity, a mid-stream rebalance would reset the connection.
Pod restarts: a graceful shutdown should finish in-flight streams (drain mode) before exiting. Kubernetes terminationGracePeriodSeconds of 60 to 120 seconds is reasonable.
Region failover: if the primary region dies mid-stream, the client gets a disconnect; client must retry to the new region, and resume support determines whether the response continues or restarts.
Anycast for SSE: works fine for the initial connection but middle-stream rerouting via BGP changes will break the TCP connection. Anycast LBs that hash on flow tuples avoid this.

Content moderation in a streaming world#

Output moderation is fundamentally incompatible with streaming: a classifier needs the full text, but the user is reading the first sentence. Practical patterns:

Input gates only: run jailbreak detection, prompt-injection classifiers, and PII filters on the prompt before the model starts. Cost: one extra 50ms call.
Streaming classifier: run a small classifier on every N tokens. If a threshold trips, abort the stream and replace the visible text with a refusal banner. Risk: the user has already seen the offending text. Useful for legal liability shielding even if not for prevention.
Post-hoc moderation: classify the full message after stream end, log violations, and use the data to retrain prompt-side filters. Does not protect this user but improves the next one.
Token-window guardrails (Anthropic, Llama Guard): process a sliding window so violations are caught a few hundred tokens late but before completion.
Hard stops: short blocklists (CSAM strings, internal credentials) can be checked per-token cheaply.

The honest framing: streaming trades safety strictness for perceived speed. Products that need strict output review (medical advice, contract clauses) should not stream those responses.

Browser UX details#

A polished streaming UI is more than appending tokens:

Typed-out animation: throttle the visible append rate to 25 to 40 characters per second so bursts do not flash by faster than the eye reads. Smooth typing animation uses a virtual buffer and requestAnimationFrame.
Scroll lock: auto-scroll to bottom while the user is at the bottom; freeze when they scroll up to re-read; resume when they hit bottom again. Use IntersectionObserver on a sentinel.
Abort button: visible from token 1, calls AbortController.abort(), fades out on done event. Server-side abort detection happens via the closed socket.
Partial markdown rendering: re-render the full message buffer through marked or remark on each chunk. Most parsers handle truncated markdown gracefully (open code fence renders as preformatted, open list as a single item).
Code-fence detection: a ``` opens a code block; before the close fence, render with a "streaming" badge. Avoid running syntax highlighter on every keystroke; debounce to every 50 ms.
Copy button stability: after stream end, the buffer is frozen; render a copy button next to the message. During stream, hide it (user clicks would copy truncated text).
Inline tool-call cards: structured-output streams render inline UI (a search result card, a calendar event) that fills in as fields arrive.
Loading state: between the request and the first token, show a typing indicator (3 dots), not a spinner. This sets the right expectation.

Real production numbers#

Number	Typical value
TTFT, cached prompt	100 to 250 ms
TTFT, uncached, 4K context	400 to 800 ms
TTFT, uncached, 100K context	2 to 6 s
Inter-token delay, Claude Opus	25 to 50 ms
Inter-token delay, GPT-4o	15 to 30 ms
Inter-token delay, Gemini 2.0 Flash	8 to 15 ms
Inter-token delay, Llama 3.1 70B (vLLM)	20 to 40 ms
End-to-end stream for 400 tokens	4 to 20 s
Abort detection latency	100 to 500 ms
WebSocket idle disconnect (corporate proxies)	30 to 120 s
nginx default `proxy_read_timeout`	60 s (raise to 600+ for long streams)
Cloudflare free tier streaming time limit	100 s per request (raise on paid)
Browser fetch timeout default	infinite, but tab eviction at ~5 min in background

Production failure modes#

Failure	Cause	Fix
Tokens arrive in one lump	CDN or proxy buffering chunked response	Set `Content-Type: text/event-stream`, `X-Accel-Buffering: no`
Stream truncates at 60 s	Default proxy read timeout	Raise `proxy_read_timeout` and ALB idle timeout
Cancel button does nothing	Server does not abort upstream on close	Wire `req.on("close")` to `AbortController.abort()`
Stream dies on tab background	Browser throttling background fetches	Use Service Worker or `keepalive`, accept the limitation
Missing cost on streamed call	Token usage only on the final event	Parse `message_delta` / `usage` events explicitly
Tool-call JSON unparseable mid-stream	Trying to JSON.parse partial input	Use a partial-JSON parser, switch to strict parse on stop
WebSocket disconnects every 60 s	Corporate proxy idle timeout	Send a ping frame every 30 s
User-visible glitch on auto-retry	EventSource reconnects but server starts a new completion	Track `Last-Event-ID`, skip already-emitted events server-side
Slow client pins GPU	Backpressure stalls upstream	Hard server timeout on idle write, abort upstream
First chunk arrives but no more	nginx flushed headers but is buffering body	Confirm `X-Accel-Buffering: no`, check `gzip` config

Quick reference#

Mental model#

Streaming hides 5 to 30 seconds of generation behind a sub-second TTFT. The model emits tokens, the transport ships deltas, the UI types out. Perceived latency is reading-bound, not generation-bound.

Transports#

Transport	Direction	Use when
SSE	server to client	default, every browser supports EventSource
WebSocket	full-duplex	client needs to cancel or talk mid-stream
Chunked HTTP + ReadableStream	server to client	custom headers needed, modern SDKs

Delta formats#

Provider	Shape
OpenAI Chat	choices[].delta.content
OpenAI Responses	response.output_text.delta typed events
Anthropic	content_block_delta with text_delta
Gemini	candidates[].content.parts[].text snapshot
Vercel AI SDK	line-prefixed: 0:"text", 9:tool, d:done
Cohere	event_type text-generation, text field

Required server headers#

Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no
Transfer-Encoding: chunked (HTTP/1.1)

CDN / proxy settings#

nginx: proxy_buffering off, proxy_read_timeout 600s
Cloudflare: Response Buffering off, paid plan for long streams
ALB: idle_timeout 4000s
HTTP/2: native DATA frames, no extra config

Abort timing budget#

Step	Latency
Client close to server detect	10 to 100 ms
Server to provider cancel	50 to 300 ms
Provider to GPU stop	50 to 200 ms (1 to 5 tokens leaked)
Total	100 to 500 ms

Retry strategies#

Fail before first token: full retry with backoff.
Fail mid-stream: best-effort partial, regenerate button.
Resumable: Anthropic Resume header with Last-Event-ID. OpenAI not yet.

JSON streaming#

Use partial-json (TS) or streaming-json (Python).
Render fields as they close.
Strict parse on content_block_stop or tool_call end.
Vercel AI SDK and OpenAI both emit incremental tool_calls arguments.

Backpressure#

TCP propagates automatically.
Slow client pins KV cache GPU memory.
Hard server timeout 30 s on idle write.
Abort upstream when client closes.

Moderation#

Input gates only for low-latency streaming.
Streaming classifier optional, late.
Post-hoc moderation for logging.
Do not stream high-stakes outputs (medical, contract).

Browser UX#

Typed-out animation 25 to 40 cps.
Auto-scroll with scroll lock at bottom.
Abort button from token 1.
Markdown re-render on each chunk (graceful with truncation).
Code-fence detection for syntax highlight.
Copy button on stream end only.
Typing indicator before first token.

Real numbers#

TTFT cached: 100 to 250 ms
TTFT uncached 4K: 400 to 800 ms
Inter-token: 10 to 50 ms
400-token completion: 4 to 20 s
Abort detect: 100 to 500 ms

Refs#

OpenAI Streaming API docs
Anthropic Messages streaming docs and Resume Events
Vercel AI SDK protocol spec
WHATWG EventSource spec
nginx proxy_buffering reference
Cloudflare Workers Streaming guide

FAQ#

How do you stream responses from an LLM?#