Skip to content

Time Synchronization#

Clock synchronization in distributed systems matters because nodes use wall-clock time to expire leases, order events, validate certificates, and pick the latest write. Crystal oscillators drift; without correction, two servers in the same rack can disagree by minutes after a few days.

flowchart TB
  GPS([GPS / atomic source]) --> S1[Stratum-1 NTP server]
  S1 --> S2[Stratum-2 server, datacenter pool]
  S2 --> H1[Host A]
  S2 --> H2[Host B]
  S2 --> H3[Host C]
  H1 -. drift over time .-> Drift([clock skew])

  classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class GPS external;
  class S1,S2,H1,H2,H3 service;
  class Drift datastore;

Four protocols cover the spectrum. NTP corrects to about 10 ms over the public internet. PTP uses hardware timestamps to reach sub-microsecond accuracy in a datacenter. TrueTime (Google Spanner) explicitly exposes a clock interval [earliest, latest] from GPS + atomic clocks and waits out the uncertainty. Hybrid logical clocks (HLC) combine a physical timestamp with a logical counter so causality is preserved even when wall clocks lie.

The system-design lesson is that wall-clock precision becomes a primary constraint as soon as you depend on it for correctness, not just for logs.

Problem statement (interviewer prompt)

Your geo-distributed database needs to assign each committed transaction a timestamp such that, if T1 commits before T2 starts, then T1's timestamp is strictly less than T2's. The clocks across your five regions are NTP-synced but can disagree by up to 50 ms. Design a scheme that gives linearizable read-after-write across regions, and discuss what changes if you have TrueTime-style bounded uncertainty hardware.

Clock synchronization in distributed systems is the discipline that keeps independent crystal oscillators close enough together that wall-clock-dependent invariants stay safe. The skew that matters in a system-design problem is not "what time is it" but "by how much can two nodes disagree, and how do I bound it."

Why nodes disagree#

Hardware clocks drift because their crystal oscillators are temperature- and voltage-sensitive. A typical server clock drifts on the order of 10 to 100 microseconds per second, or roughly 1 second per day uncorrected. Virtualisation, NUMA migrations, and CPU frequency scaling make it worse. Without correction, two machines in the same rack diverge by minutes after a week.

NTP: the workhorse#

The Network Time Protocol corrects clocks against a hierarchy of servers. A stratum-0 source (GPS, atomic clock) feeds stratum-1 servers; the public pool is stratum-2 or 3.

sequenceDiagram
  participant C as Client
  participant S as NTP server
  Note over C: t1 = local time of send
  C->>S: NTP request (t1)
  Note over S: t2 = receive time, t3 = transmit time
  S->>C: response (t1, t2, t3)
  Note over C: t4 = receive time, offset = ((t2-t1)+(t3-t4))/2, rtt = (t4-t1)-(t3-t2)

The client samples many servers, throws out outliers, and slews (gradually adjusts rate) or steps (jumps) the local clock. Practical accuracy on the public internet: 10 ms. Inside a datacenter with chrony and stratum-1 servers: 1 ms.

NTP can run backwards on big steps; this breaks any code that assumes monotonic time. Always read monotonic time (CLOCK_MONOTONIC on Linux, Instant in Java) for timeouts and durations; reserve wall-clock (CLOCK_REALTIME) for timestamps shown to humans or stored alongside data.

PTP: when microseconds matter#

The Precision Time Protocol (IEEE 1588) hardware-timestamps packets at the NIC, so it sidesteps OS scheduling jitter. With boundary clocks in every switch, you reach sub-microsecond accuracy across a datacenter. PTP underpins financial-exchange clocks, telco 5G, and AWS's Time Sync Service.

flowchart LR
  M[PTP grandmaster<br/>GPS-disciplined] --> SW1[Boundary clock switch]
  SW1 --> SW2[Boundary clock switch]
  SW2 --> H1[Server NIC<br/>hardware timestamp]
  SW2 --> H2[Server NIC<br/>hardware timestamp]

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
  class M external;
  class SW1,SW2,H1,H2 service;

TrueTime: API-level uncertainty#

Spanner's TrueTime exposes the clock as an interval, not a point:

TT.now() = { earliest: T - epsilon, latest: T + epsilon }

epsilon is bounded by deploying GPS receivers and atomic clocks on every Spanner node; the published bound is typically under 7 ms. Two primitives are then trivial:

  • TT.after(t) : now.earliest > t
  • TT.before(t) : now.latest < t
sequenceDiagram
  participant T as Transaction
  participant TS as Timestamp service
  participant TT as TrueTime
  T->>TS: prepare commit
  TS->>TT: pick commit ts = now.latest
  Note over TS: commit wait until TT.after(commit_ts)
  TS-->>T: ack commit

The commit wait is the key trick: the transaction waits until the clock has provably advanced past the commit timestamp before acknowledging. After that wait, any later transaction will pick a strictly larger timestamp, giving externally consistent (linearizable) order across the entire fleet at single-digit-ms cost.

Hybrid logical clocks#

When you cannot afford GPS + atomic clocks, HLCs (Kulkarni & Demirbas, 2014) give you the next best thing: a 64-bit timestamp that is bounded close to physical time and monotonic with causality.

import time

class HLC:
    def __init__(self):
        self.l = 0    # logical wall component (microseconds since epoch)
        self.c = 0    # logical counter

    def _now_pt(self) -> int:
        return int(time.time() * 1_000_000)

    def send(self) -> tuple[int, int]:
        pt = self._now_pt()
        if pt > self.l:
            self.l, self.c = pt, 0
        else:
            self.c += 1
        return self.l, self.c

    def receive(self, l_msg: int, c_msg: int) -> tuple[int, int]:
        pt = self._now_pt()
        new_l = max(self.l, l_msg, pt)
        if new_l == self.l == l_msg:
            new_c = max(self.c, c_msg) + 1
        elif new_l == self.l:
            new_c = self.c + 1
        elif new_l == l_msg:
            new_c = c_msg + 1
        else:
            new_c = 0
        self.l, self.c = new_l, new_c
        return self.l, self.c

CockroachDB uses HLC for transaction timestamps under a configured max_offset (default 500 ms): any read whose timestamp falls inside the uncertainty window must restart at a later one, which preserves serialisability without TrueTime hardware.

What clock skew breaks#

Skew assumption Failure when violated
TLS / JWT validity (notBefore / exp) Connections rejected or accepted incorrectly; tokens replayable
Lease expiry (expire_at < now) Two leaders, split-brain
Cookie / session expiry Premature logout or extended attacker windows
Database last-write-wins Older write overwrites newer; data loss
Cron schedules Missed runs or duplicate runs
Distributed tracing latency math Spans appear to take negative time

Lease-based designs (Chubby, etcd) build in a safety margin: a leader treats its own lease as expired delta seconds before the holder's view, where delta exceeds the worst-case skew.

Picking a protocol#

Need Protocol
"Logs roughly align across servers" NTP (chrony, ntpsec)
"Microsecond ordering across the datacenter" PTP with boundary-clock switches
"External consistency across regions" TrueTime-style API + commit-wait
"Causally correct timestamps on commodity hardware" HLC + max_offset
"Monotonic timeouts" Always CLOCK_MONOTONIC / Instant

These are layered, not competing: most fleets run NTP for the boot clock, switch to PTP for low-latency workloads, and add HLC at the application layer.

Operational pitfalls#

  • Asymmetric network paths break NTP offset estimation; multipath leaves the offset biased.
  • Leap seconds cause clock_realtime to repeat; smear them (Google leap smear) or your timer code will be unhappy.
  • VM live migration can step the clock; pin time-sensitive workloads or use the hypervisor's paravirt clock.
  • Trusting time.time() for security: tokens, signatures, and audit logs need a defended source.
  • Failing to monitor offset and dispersion: when NTP silently goes off the rails, you find out only when a lease misbehaves.

Where time-sync fits#

flowchart TB
  TS((Time<br/>synchronization))
  LC[Logical clocks<br/>Lamport, vector, HLC]
  CON[Consensus Raft Paxos<br/>leases depend on skew]
  DTX[Distributed transactions<br/>commit-wait, snapshot ts]
  TS --> LC
  TS --> CON
  TS --> DTX

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  class TS service;
  class LC,CON,DTX datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Logical clocks Lamport / vector / HLC, the causality primitives logical-clocks
HLD Consensus Raft Paxos leaders rely on lease expiry; skew threatens safety consensus-raft-paxos
HLD Distributed transactions Spanner / Cockroach use TrueTime / HLC for commit ordering distributed-transactions

Quick reference#

Why it matters#

Wall-clock skew can break leases, ordering, certificates, and last-write-wins. Bound the skew, or design without depending on it.

Drift baseline#

Server crystal oscillators drift ~1 second per day uncorrected. VM migration and frequency scaling make it worse.

Protocols#

Protocol Accuracy Notes
NTP ~10 ms (internet), ~1 ms (LAN with chrony) Hierarchical, stratum 0..15; sample many servers, reject outliers
PTP (IEEE 1588) sub-microsecond NIC hardware timestamps, boundary-clock switches
TrueTime (Spanner) bounded interval, epsilon ~7 ms GPS + atomic on every node, API exposes now.earliest, now.latest
HLC bounded close to physical, monotonic w/ causality Software-only; CockroachDB uses it with max_offset

Read the right clock#

  • Timeouts and durations: monotonic (CLOCK_MONOTONIC, Instant)
  • Human timestamps and DB columns: wall (CLOCK_REALTIME)
  • NTP can step CLOCK_REALTIME backwards; never use it for timers

TrueTime trick: commit-wait#

  • Pick commit_ts = TT.now().latest
  • Wait until TT.after(commit_ts) before acking commit
  • Guarantees any later transaction picks a strictly larger timestamp -> external consistency

HLC update rule (sketch)#

  • On send: l = max(l, pt), if unchanged bump c
  • On receive: take max(l_local, l_msg, pt), bump c per the tiebreak rule

What skew breaks#

  • TLS / JWT validity windows
  • Lease expiry (split brain in leader election)
  • LWW conflict resolution (data loss)
  • Cron triggers (missed / duplicated)
  • Trace latency math (negative spans)

Mitigations#

  • Build slack into leases (delta greater than worst skew)
  • Use logical clocks for causality, not wall time
  • Smear leap seconds (Google leap smear)
  • Monitor NTP offset, dispersion, stratum

Pitfalls#

  • Asymmetric network paths bias NTP offset
  • VM live migration steps the clock
  • Trusting time.time() for security checks
  • Forgetting to set max_offset in HLC systems

Refs#

  • IETF RFC 5905 - NTPv4
  • IEEE 1588-2019 - PTP
  • Corbett et al. - "Spanner: Google's Globally-Distributed Database" (OSDI 2012)
  • Kulkarni, Demirbas - "Logical Physical Clocks" (HLC paper, 2014)
  • CockroachDB - max_offset and uncertainty intervals

FAQ#

Why does clock synchronization matter in distributed systems?#

Nodes use wall-clock time for leases, certificate expiry, log ordering, and last-write-wins. Untrusted clocks can violate ordering, expire credentials early, or break safety invariants.

How accurate is NTP?#

Public NTP delivers tens of milliseconds across the internet and below one millisecond on a clean LAN. It is good enough for most workloads but not for sub-microsecond ordering.

When should I use PTP instead of NTP?#

Use Precision Time Protocol on a LAN that needs microsecond-level accuracy, like high-frequency trading, telecom, or distributed databases that bound transaction ordering on clock error.

What is TrueTime in Spanner?#

An API that returns an interval bounding real time, derived from GPS and atomic clocks. Spanner waits out this uncertainty after a write so global commit ordering is correct.

What are hybrid logical clocks?#

Timestamps that combine physical time with a monotonic logical counter so they always move forward. They give human-readable values and preserve causality despite clock drift.