Object Storage (S3-like)#

Problem statement (interviewer prompt)

Design an S3-compatible object storage system: durable, eventually unlimited capacity, with PUT/GET/DELETE/LIST by key, multipart upload, versioning, lifecycle transitions to colder tiers, server-side encryption, and 11×9s durability via erasure coding.

flowchart LR
  C([Client])
  GW[REST API Gateway]
  META[(Metadata index<br/>key -> blob loc)]
  BLOB[(Blob nodes<br/>erasure coded)]
  C -->|PUT/GET key| GW
  GW --> META
  GW --> BLOB

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C client;
    class GW edge;
    class META,BLOB storage;

flowchart TB
  subgraph Client[Clients]
    SDK([SDK / CLI])
    BR([Browser pre-signed])
    APP[App service]
  end

  subgraph Edge[Edge]
    DNS[Regional endpoint DNS]
    LB[L7 LB]
    GW[REST Frontend<br/>SigV4 auth, range, multipart]
  end

  subgraph Control[Control Plane]
    AUTH[IAM / SigV4 verify]
    POL[Bucket policy / ACL]
    LIFECYCLE[Lifecycle rules<br/>tier transition / expiry]
    VERSION[Versioning manager]
    REPL[Cross-region replication]
    EVENTS[[Event notifications<br/>SNS / Lambda]]
  end

  subgraph Index[Metadata / Index]
    KEYS[(Key index<br/>distributed B-tree)]
    PART[Partitioning by key prefix]
    HOT[Hot-shard splitter]
  end

  subgraph Data[Data Plane - Storage Nodes]
    PLACE[Placement service]
    SHARD[Erasure coding<br/>e.g. 10+4 RS]
    CHUNK[Chunks 4-64 MB]
    NODE1[Storage node 1]
    NODE2[Storage node 2]
    NODE3[Storage node 3]
    NODEN[Storage node N]
    SCRUB[Scrubber / repair]
    BG[Background rebalance]
  end

  subgraph Tiering
    STANDARD[Standard tier - SSD/HDD mix]
    IA[Infrequent access]
    GLACIER[Cold / Glacier / Archive<br/>tape, minutes-to-hours retrieval]
  end

  subgraph Features
    MULTI[Multipart upload]
    PRES[Pre-signed URLs]
    CMK[KMS / SSE / CSE encryption]
    LOCK[Object lock / WORM / Legal hold]
    LOG[Server access logs]
    RR[Requester-pays]
    SELECT[S3 Select / pushdown]
  end

  Client --> DNS --> LB --> GW --> AUTH --> POL
  GW --> KEYS
  GW --> PLACE --> SHARD --> CHUNK
  CHUNK --> NODE1
  CHUNK --> NODE2
  CHUNK --> NODE3
  CHUNK --> NODEN
  SCRUB -.repair.-> Data
  BG -.move.-> Data
  LIFECYCLE -.transition.-> Tiering
  VERSION --- KEYS
  REPL -.async.-> Data
  EVENTS -.fire.-> Client
  Features --- GW

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class SDK,BR client;
    class DNS,LB edge;
    class APP,GW,AUTH,LIFECYCLE,VERSION,REPL,PART,HOT,PLACE,SHARD,CHUNK,NODE1,NODE2,NODE3,NODEN,SCRUB,BG,STANDARD,IA,MULTI,PRES,CMK,LOCK,RR service;
    class KEYS,GLACIER datastore;
    class EVENTS queue;
    class POL,SELECT storage;
    class LOG obs;

Architecture beats#

Flat keyspace keyed by bucket + object_key; no real directories.
Metadata (index) is separate from data (blobs). Index is itself a distributed sorted structure.
Erasure coding: each object split into chunks; chunks encoded into N+K shards (e.g. Reed-Solomon 10+4) across nodes/racks/AZs.
Durability of 11×9s comes from EC + cross-AZ placement + scrubbing.

Consistency#

Modern S3: strong read-after-write across all regions for new objects.
Historically eventually consistent for overwrites/deletes.

Multipart upload#

Initiate → upload parts (parallel) → complete (server assembles).
Allows resumable uploads of TB-scale objects.

Lifecycle & tiering#

Transition rules: > 30 d to IA, > 90 d to Glacier, > 365 d delete.
Restore from cold tiers takes minutes to hours; pay-per-restore.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	Load balancer / GSLB	L4/L7 traffic distribution and failover	load-balancer
`HLD`	Sharding	horizontal partitioning across nodes	database-sharding
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	Leader/follower replication	sync/semi-sync/async replication, failover	replication-leader-follower
`HLD`	LSM vs B-Tree engines	WAL, memtable, SSTables, compaction	storage-engines-lsm-btree
`HLD`	Realtime protocols	WS / SSE / polling / gRPC streaming	realtime-protocols
`HLD`	Multi-region & DR	RTO / RPO, active-active, failover	multi-region-dr
`LLD`	REST API design	verbs, statuses, pagination, errors	rest-api-design

Quick reference#

Functional#

PUT / GET / DELETE / LIST by key.
Multipart upload + resumable.
Versioning, lifecycle transitions, object lock.
Server-side / client-side encryption.
Cross-region replication.
Event notifications.

Non-functional#

11×9 durability.
99.99% availability per region.
Strong read-after-write consistency.
Effectively unlimited capacity.

Capacity#

S3 reportedly hosts trillions of objects, exabytes total.
Per-bucket: no hard limit; sharding by key prefix handles hot ranges.

API#

PUT /{bucket}/{key}            (Object body)
GET /{bucket}/{key}            (Range supported)
DELETE /{bucket}/{key}
POST /{bucket}/{key}?uploads    (multipart init)

Data model#

bucket(name, region, policy, lifecycle)
object(bucket, key, version_id, etag, size, mtime, storage_class, encryption)
Storage: chunked + EC across nodes.

Trade-offs#

Flat keyspace scales but listing by prefix can become a hot spot - randomize prefixes for high-throughput buckets.
Erasure coding vs replication: EC saves storage (~1.4× overhead) but rebuilds cost CPU/network on failure.
Strong consistency needed adapting metadata layer; older S3 used to be eventually consistent.

Refs#

AWS S3 white papers, "Building S3" re:Invent talks.
GCS architecture papers; Ceph RADOS paper.
Reed-Solomon coding; FAST conference papers on object storage.

FAQ#

How does S3 achieve 11 nines of durability?#

Objects are erasure coded into data and parity shards spread across multiple availability zones. Background scrubbing detects bit rot and rebuilds shards before too many failures accumulate.

What is the role of the metadata service in object storage?#

Metadata maps each key to the locations of its erasure shards plus version, ACL, and checksum info. It is a heavily sharded, replicated index designed for high read throughput.

How does multipart upload work in S3?#

Clients split large objects into parts, upload each in parallel, then issue a complete request listing the part ETags. The server stitches them into one object atomically on completion.

Why is S3 eventually consistent for some operations?#

Strong consistency now applies to reads after writes, but list operations on prefixes can lag briefly because the bucket index is sharded and replicated asynchronously across regions.

How does S3 implement lifecycle transitions to colder storage?#

Bucket lifecycle rules schedule background jobs to transition objects to Infrequent Access, Glacier, or Deep Archive based on age. Each tier trades retrieval latency and cost for storage cost.

Video walkthrough

Object Storage in System Design Interviews : via Hello Interview