TCP Deep Dive#

Problem statement (interviewer prompt)

Explain how TCP guarantees in-order, reliable delivery over an unreliable IP network. Cover the handshake, retransmission, flow control, and congestion control mechanisms that make a long-running stream resilient to packet loss and varying bandwidth.

TCP is a connection-oriented, reliable, ordered byte stream over IP. Every byte gets a sequence number; every byte gets acknowledged; the sender retransmits anything not acked within an RTT-estimated timeout.

sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: SYN, seq=x
  S->>C: SYN-ACK, seq=y, ack=x+1
  C->>S: ACK, ack=y+1
  Note over C,S: connection established
  C->>S: data + seq numbers
  S->>C: ACK
  C->>S: FIN
  S->>C: FIN-ACK

Four pillars: handshake (establishes seq numbers), reliable transfer (retransmit + ACK), flow control (receiver window), congestion control (sender estimates network capacity).

TCP turns an unreliable, out-of-order IP datagram service into a reliable, ordered byte stream. Understanding the moving parts explains nearly every TCP-related performance issue.

Connection lifecycle#

stateDiagram-v2
  [*] --> CLOSED
  CLOSED --> SYN_SENT : client connect()
  CLOSED --> LISTEN : server bind+listen
  LISTEN --> SYN_RCVD : SYN received
  SYN_SENT --> ESTABLISHED : SYN-ACK + ACK
  SYN_RCVD --> ESTABLISHED : ACK
  ESTABLISHED --> FIN_WAIT_1 : close()
  ESTABLISHED --> CLOSE_WAIT : peer FIN
  FIN_WAIT_1 --> TIME_WAIT : FIN-ACK
  TIME_WAIT --> CLOSED : 2 MSL
  CLOSE_WAIT --> LAST_ACK : close()
  LAST_ACK --> CLOSED : ACK

TIME_WAIT (2 × Maximum Segment Lifetime, typically 60 seconds) protects against delayed packets from the old connection corrupting a fresh one on the same port pair.

Reliable transfer#

Every TCP segment carries a sequence number (byte offset, not packet count). The receiver acks next_expected_byte:

sender:   [seq=1000, 100 bytes]
receiver: ACK 1100   # I've got everything < 1100

Loss detection#

Mechanism	Trigger
Retransmission timeout (RTO)	No ACK within smoothed RTT × factor
Fast retransmit	3 duplicate ACKs ("dupacks") indicate a gap
SACK (selective ACK)	Receiver tells sender exactly which ranges arrived

RTO doubles on each loss (Karn's algorithm). SACK + fast retransmit dramatically beat RTO-only behaviour on lossy links.

Flow control - the receiver window#

The receiver advertises a WIN field on every ACK:

ACK 5000  WIN=64KB    # I have 64KB of buffer space

The sender never sends more than WIN unacked bytes. Classic bug: receiver app reads slowly, window shrinks to zero, throughput collapses.

Congestion control - the sender's estimate#

The sender maintains a cwnd (congestion window). It can send at most min(cwnd, rwnd) unacked bytes.

flowchart TB
  SS[Slow Start<br/>cwnd doubles each RTT] --> CA[Congestion Avoidance<br/>cwnd += 1 MSS per RTT]
  CA -- loss --> Reduce[cwnd /= 2]
  Reduce --> CA

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class SS,CA,Reduce service;

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;

Algorithm families#

Algo	Style	Use
Reno / NewReno	Loss-based; cwnd halves on loss	Legacy default
CUBIC	Loss-based; cubic curve recovery	Linux 2.6.19+ default
BBR	Bandwidth-delay product; ignores loss	Google, YouTube, Cloudflare
Vegas	Latency-based	Research
DCTCP	ECN-aware	Datacentre networks

BBR is the major modern shift: it models the bottleneck bandwidth and minimum RTT, then paces packets accordingly. It outperforms CUBIC on links with non-congestion loss (wifi, cell).

Throughput formula#

throughput ≈ min(cwnd, rwnd) / RTT

Doubling RTT halves throughput. This is why CDNs matter; halving RTT doubles throughput.

Nagle's algorithm#

if there is unacked data:
    buffer small writes; send only when 1 MSS available or all data acked

Reduces tinygram-induced congestion but adds latency. Disable for chatty interactive protocols (gRPC, ssh) with TCP_NODELAY.

Common performance issues#

Symptom	Likely cause
Stair-step throughput	Receiver app reading slowly; window pressure
Slow first byte	TCP slow start; cold connection
Tail-latency spikes	Retransmissions on lossy link
1 MB transfer takes >1s	RTT × slow-start ramp
Lots of TIME_WAIT	High connection churn (use keep-alive)

Tuning knobs#

tcp_window_scaling and tcp_rmem / tcp_wmem (Linux): allow large windows for high BDP links.
tcp_no_metrics_save = 0: reuse RTT/cwnd estimates across connections.
net.core.somaxconn: server accept-queue depth.
SO_REUSEPORT: spread accept across multiple processes.
TCP_FASTOPEN: skip handshake RTT on warm connections.

Where TCP sits in the stack#

flowchart TB
  TCP((TCP))
  NET[Network basics<br/>parent overview]
  H2[HTTP/2 deep dive<br/>multiplexed on TCP]
  UDP[UDP use cases<br/>alternative transport]
  H3[HTTP/3 + QUIC<br/>TCP replacement]
  NET --> TCP
  TCP --> H2
  TCP -. alternative .- UDP
  H3 -. replaces .- TCP

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class TCP service;
    class NET,H2,UDP,H3 datastore;

Glossary & fundamentals#

Tag	Concept	What it is	Page
`HLD`	Network basics	the OSI stack overview	network-basics
`HLD`	HTTP2 deep dive	HTTP/2 over TCP	http2-deep-dive
`HLD`	UDP use cases	the alternative	udp-use-cases
`HLD`	HTTP3 QUIC	TCP replacement	http3-quic

Quick reference#

Handshake#

SYN → SYN-ACK → ACK; 1 RTT before any data.

Close#

FIN → ACK → FIN → ACK; TIME_WAIT for 2 MSL (~60s).

Reliable delivery#

Sequence numbers on every byte
ACK = next-expected-byte
Loss = RTO (timeout) or 3 dupacks (fast retransmit)
SACK reports actual received ranges

Flow control#

Receiver advertises WIN; sender never exceeds it. Window shrinks if app reads slowly.

Congestion control#

Algo	Use
Reno/NewReno	Legacy
CUBIC	Linux default
BBR	Google, Cloudflare; bandwidth-modeled
Vegas	Latency-based; research
DCTCP	Datacentre, ECN-aware

Loss → cwnd halves; recovery rebuilds via slow-start then congestion-avoidance.

Throughput#

min(cwnd, rwnd) / RTT

Tuning#

tcp_window_scaling for high BDP
TCP_NODELAY to disable Nagle
TCP_FASTOPEN to skip handshake RTT
SO_REUSEPORT for accept fan-out

Watch-outs#

Slow first byte = slow start
Lots of TIME_WAIT = poor keep-alive
Tail latency on wifi = retransmissions
Bufferbloat = oversized router buffers

Refs#

RFC 9293
Cardwell et al. BBR paper
Stevens - TCP/IP Illustrated v1

FAQ#

How does the TCP three-way handshake work?#

Client sends SYN with sequence x, server replies SYN-ACK with seq y and ack x+1, then client ACKs y+1. After this exchange both sides agree on initial sequence numbers.

What is the difference between TCP flow control and congestion control?#

Flow control protects the receiver via the advertised window. Congestion control protects the network by adjusting the sender's congestion window based on loss or RTT signals.

What is CUBIC vs BBR in TCP?#

CUBIC is loss-based and probes by growing the window until it sees drops. BBR models bandwidth and RTT to send at the bottleneck rate, performing better on lossy links.

Why does TCP suffer head-of-line blocking?#

TCP delivers a single ordered byte stream, so if one segment is lost the kernel holds back later segments until retransmission, stalling unrelated higher-level streams.

When should I use TCP over UDP?#

Use TCP when you need reliable, ordered delivery without writing your own retransmit logic: HTTP, SSH, databases, file transfer. Use UDP when you tolerate loss for latency.

Network Basics: the OSI/IP overview TCP sits within
HTTP/2 Deep Dive: the highest-throughput TCP use case
HTTP/3 and QUIC: the TCP-replacement story