Document Database (MongoDB-style)

Problem statement (interviewer prompt)

Design a MongoDB-style document database: BSON documents grouped into collections, rich query language with secondary indexes, multi-document transactions, replica sets, sharding by a chosen shard key, and change streams (CDC).

flowchart LR
  C[App]
  R[mongos / Router]
  S1[(Shard 1<br/>replica set)]
  S2[(Shard 2<br/>replica set)]
  S3[(Shard 3<br/>replica set)]
  CS[(Config servers<br/>shard map)]
  C --> R
  R --> S1
  R --> S2
  R --> S3
  R --- CS

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class C service;
    class R,S1,S2,S3,CS datastore;

flowchart TB
  subgraph Client[App tier]
    APP[App]
    DRV([Smart driver<br/>topology aware])
  end

  subgraph Routing[Routing - mongos]
    MONGOS[Router]
    CFG[(Config servers<br/>3-5 Raft / RS)]
    BAL[Balancer<br/>chunk migration]
  end

  subgraph Shard1[Shard 1 - replica set]
    P1[Primary]
    S1A[Secondary]
    S1B[Secondary]
    ARB1[Arbiter]
  end

  subgraph Shard2[Shard 2 - replica set]
    P2[Primary]
    S2A[Secondary]
    S2B[Secondary]
  end

  subgraph Shard3[Shard 3 - replica set]
    P3[Primary]
    S3A[Secondary]
    S3B[Secondary]
  end

  subgraph Engine[Storage engine - WiredTiger]
    BPTREE[B-Tree default]
    LSMOPT[LSM optional]
    MMAP[mmap legacy]
    CACHE[Page cache]
    JOURNAL[WAL / Journal]
    SNAP[MVCC snapshots]
  end

  subgraph Repl[Replication]
    OPLOG[Oplog idempotent ops]
    HEARTB[Heartbeats every 2s]
    ELECT[Election via Raft]
    READPREF[Read preferences<br/>primary / nearest / secondary]
    WC[Write concern w-majority]
    RC[Read concern majority / linearizable]
  end

  subgraph Index[Indexing]
    BIDX[B-tree single field]
    COMP[Compound]
    TEXT[Text index]
    GEO[2dsphere geo]
    PART[Partial]
    HASHED[Hashed]
    TTL[TTL]
    VEC[Vector / Atlas Search]
  end

  subgraph CDC[Change Streams]
    OPL2[Oplog tail]
    CHG[Change Streams API]
    DOWN[Downstream: search, cache, analytics]
  end

  subgraph Features
    AGG[Aggregation framework]
    JOIN[$lookup join]
    TX[Multi-doc transactions<br/>snapshot isolation across shards]
    GRIDFS[GridFS - large files]
  end

  APP --> DRV --> MONGOS
  MONGOS --- CFG
  MONGOS --> Shard1
  MONGOS --> Shard2
  MONGOS --> Shard3
  BAL -.chunk move.-> Shard1
  P1 -. oplog .-> S1A
  P1 -. oplog .-> S1B
  P2 -. oplog .-> S2A
  P3 -. oplog .-> S3A
  Engine --- Shard1
  Repl --- Shard1
  Index --- Shard1
  OPL2 --- P1
  OPL2 --> CHG --> DOWN

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DRV client;
    class APP,MONGOS,BAL,P1,S1A,S1B,ARB1,P2,S2A,S2B,P3,S3A,S3B,BPTREE,LSMOPT,MMAP,JOURNAL,SNAP,OPLOG,HEARTB,ELECT,READPREF,WC,RC,BIDX,COMP,TEXT,GEO,PART,HASHED,TTL,VEC,OPL2,CHG,DOWN,AGG,JOIN,TX,GRIDFS service;
    class CFG datastore;
    class CACHE cache;

Sharding

• Shard key chosen at collection level (hashed or ranged).
• Data split into chunks (~64-128 MB); balancer migrates chunks for even load.
• Cross-shard queries scatter-gather; targeted queries hit one shard if shard key supplied.

Replication & elections

• Replica set = 3 to 7 nodes; one primary, rest secondaries.
• Oplog is a capped collection containing idempotent ops.
• Elections via Raft-like protocol; replica with highest oplog wins.

Transactions

• Multi-document, multi-collection transactions across shards (since 4.2).
• Snapshot isolation; cost is non-trivial compared to single-doc updates.

Indexes

• Lots of types; choice dominates query latency.
• Indexes are per-shard; a global secondary index requires app-level work.

Change Streams (built-in CDC)

• Tails the oplog and exposes a typed cursor.
• Use for cache invalidation, search sync, downstream services.

Glossary & fundamentals

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
HLD	Sharding	horizontal partitioning across nodes	database-sharding
HLD	Cache strategies	cache-aside, read/write-through, eviction	caching-strategies
HLD	CAP / PACELC	C vs A under partition; L vs C otherwise	cap-pacelc
HLD	Raft / Paxos consensus	replicated state machine via majority quorum	consensus-raft-paxos
HLD	Leader/follower replication	sync/semi-sync/async replication, failover	replication-leader-follower
HLD	LSM vs B-Tree engines	WAL, memtable, SSTables, compaction	storage-engines-lsm-btree
HLD	MVCC & isolation levels	snapshot isolation, serializability, vacuum	mvcc-isolation-levels
HLD	Change Data Capture	WAL/binlog tailing, outbox publishing	change-data-capture
LLD	REST API design	verbs, statuses, pagination, errors	rest-api-design

Quick reference

Functional

• Schema-flexible JSON / BSON documents.
• Rich query language with secondary indexes.
• Sharding + replication built-in.
• Aggregation pipelines.
• Multi-document transactions.

Non-functional

• 10-50k ops/s/shard (varies with workload).
• Replica set HA: failover seconds.
• Linearizable reads on request (read concern majority + linearizable).

Capacity

• Per shard: 1-4 TB working set; up to 64 TB+ disk.
• Cluster sizes routinely 10-100 shards.

API

• Drivers (Mongo, Couchbase) speak native protocol; OpQuery / OP_MSG.
• HTTP layer via Atlas Data API or REST gateways.

Schema (conceptual)

• db.collection holds documents {_id, ...}.
• Shard key chosen at create time, hard to change.

Trade-offs

• Flexible schema = velocity at start, debt over time. Add JSON schema validators.
• Embedded vs referenced docs: embed for 1-to-few; reference for many-to-many.
• Sharded transactions are real but slow; design hot paths to stay single-shard.
• Eventually consistent secondary reads vs majority reads: pick per query.

Refs

• MongoDB docs (sharding, replication, oplog, change streams).
• "Designing Data-Intensive Applications" doc DB chapter.
• Couchbase architecture papers; Azure Cosmos DB internals.

FAQ

How is a document database different from a relational database?

Document databases store nested BSON or JSON documents grouped into collections, with flexible per document schemas. Relational databases enforce a fixed schema and join across normalized tables.

What is a shard key in MongoDB?

The shard key is the field used to partition documents across shards. A good shard key spreads writes evenly, supports common queries with targeted routing, and avoids hot keys.

How does a replica set provide high availability?

A replica set has one primary and multiple secondaries. Writes go to the primary and replicate asynchronously. If the primary fails, the set elects a new primary via a Raft-style protocol.

How do change streams work in a document database?

Change streams expose the replication oplog as a real time feed of inserts, updates, and deletes. Applications subscribe to drive CDC pipelines, search indexes, and event driven workflows.

When should you choose a document database over SQL?

Document databases shine when the schema is evolving, data is naturally hierarchical, and most queries fetch a single aggregate. Pick SQL when you need rich joins, strict constraints, or complex transactions.