Skip to content

GitHub-Style Source Hosting#

Problem statement (interviewer prompt)

Design a multi-tenant Git hosting platform like GitHub or GitLab: serve git clone/push over HTTPS+SSH, expose a web UI for browse / PR / review, run CI hooks, full-text code search across millions of repos, and webhooks to third parties.

flowchart TB
  Dev[Developer] --> Edge[Edge LB + auth]
  Edge --> Git[Git protocol server]
  Edge --> Web[Web UI / API]
  Git --> Storage[(Repo storage<br/>sharded git filesystem)]
  Web --> Meta[(Metadata DB<br/>repos, users, PRs)]
  Web --> Search[(Code search index)]
  Edge -->|webhook on push| Hook[Hook dispatcher]
  Hook --> Q[(Webhook queue)]
  Q --> Out[Outbound to CI / Slack / Jira]
  Web --> Pages[Static pages CDN]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Dev client;
    class Edge,Pages edge;
    class Git,Web,Hook service;
    class Storage,Meta,Search datastore;
    class Q queue;
    class Out external;

Two distinct traffic shapes: Git protocol (huge transfers, long-lived) and web/API (short requests). Different fleets.

A multi-tenant Git host serves three distinct workloads on top of the same data: Git protocol, web/API, and search. Each has its own fleet and storage path.

Repo storage and sharding#

flowchart LR
  Req[Repo request<br/>org/repo] --> Resolver[Shard resolver]
  Resolver --> S1[Storage shard 1<br/>git filesystems]
  Resolver --> S2[Storage shard 2]
  Resolver --> S3[Storage shard 3]
  S1 -.replica.-> R1[Read replica]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Resolver service;
    class S1,S2,S3,R1 datastore;

Each shard is a fleet of nodes running real git against a local filesystem. Resolver maps org/repo to a shard (consistent hashing).

Replicas: usually 3-way for durability + read scale. Cross-DC for HA.

Git protocol fleet#

Git over HTTPS / SSH. Connections are long (full clones can take minutes); receives need to forward the packfile to storage and update refs atomically.

sequenceDiagram
  participant Dev
  participant LB as Edge LB
  participant Git as Git protocol pod
  participant Stor as Storage shard
  Dev->>LB: git push (HTTPS)
  LB->>Git: route by org/repo
  Git->>Stor: stream pack to shard primary
  Stor->>Stor: receive-pack hook (validate, run policy)
  Stor->>Stor: update refs atomically
  Stor-->>Git: ok
  Git-->>Dev: 200

PR / code review service#

flowchart TB
  Push --> Hook[Pre/post hook]
  Hook --> PR[(PR service)]
  PR --> Diff[Diff renderer]
  PR --> Review[Review threads]
  PR --> Check[Check runs CI / status]
  PR --> Merge[Merge strategies: merge / squash / rebase]
  Merge --> Stor[Storage shard]

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Hook,PR,Diff,Review,Check,Merge service;
    class Stor datastore;

PR has its own metadata (title, description, reviewers, state) plus a list of commit hashes from the underlying Git refs.

Code search at scale#

flowchart LR
  Push --> Idx[Indexer]
  Idx --> Search[(Code search index<br/>Zoekt / Hound / GitHub Blackbird)]
  Q[User query] --> Search
  Search --> Results

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class Idx service;
    class Search,Results datastore;

Trigram indexes (Zoekt, Hound) for fast regex search over millions of repos. GitHub's "Blackbird" indexes 200B+ tokens with sub-second response.

Webhooks#

Async; per-org per-event configurable. Retries on 5xx; deliver attempt log visible to admin.

Abuse defences#

  • Per-IP and per-account rate limits.
  • Push protection (secret scanning) blocks pushes with detected secrets.
  • DMCA takedown workflow.
  • Anti-abuse heuristics for fake accounts.

Capacity sketch#

GitHub: 100M+ devs, 400M+ repos, 1B+ commits/year. Reads dominate: 99% of Git protocol traffic is clone/fetch.

How GitHub-style hosting composes#

flowchart TB
  GH((GitHub-style<br/>source hosting))
  GIT[Git VCS<br/>object model]
  DBS[Database sharding<br/>repo sharding]
  SRCH[Search internals<br/>code search index]
  WH[Webhooks<br/>outbound delivery]
  GIT --> GH
  GH --> DBS
  GH --> SRCH
  GH --> WH

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class GH service;
    class GIT,DBS,SRCH,WH datastore;

Glossary & fundamentals#

Tag Concept What it is Page
HLD Git VCS the underlying storage git-vcs
HLD Database sharding repo-level sharding database-sharding
HLD Search internals code-search index search-internals
HLD Webhooks the outbound delivery layer webhooks

Quick reference#

Functional#

  • Git protocol over HTTPS / SSH
  • Web UI for browse / PR / review
  • Issue tracking
  • Code search across all repos
  • Webhooks to third parties
  • Org / team RBAC

Non-functional#

  • Push p99 < 5s for typical repo
  • Clone p99 < 30s
  • Web UI p99 < 200ms
  • 99.95% availability

Repo storage#

  • Shard by org/repo (consistent hash)
  • 3-way replicas across DC
  • Real git filesystems on each shard

Distinct fleets#

  • Git protocol pods (long connections, big transfers)
  • Web/API pods (short requests)
  • Search indexer + serving
  • Webhook dispatch

Trigram indexes (Zoekt, Hound, GitHub Blackbird) at billions of tokens.

Webhooks#

Per-event configurable; retry on 5xx; per-org rate limit on outbound.

Capacity#

GitHub-scale: 100M+ devs, 400M+ repos. Reads >>> writes (clone/fetch dominate).

Abuse defences#

  • Push protection / secret scanning
  • Per-IP and per-account rate limits
  • DMCA workflow
  • Anti-spam heuristics

Refs#

  • GitHub Blackbird code search blog
  • GitHub DGit (sharded storage) blog
  • GitLab architecture docs

FAQ#

How do you design a GitHub-style hosting platform?#

Run a sharded Git filesystem behind a Git protocol server for clones and pushes, a separate web/API tier for UI and PRs, a code search index, and an outbound webhook dispatcher.

How does GitHub scale Git storage?#

Repositories are sharded across many file servers by repo ID. A routing layer maps a request URL to the shard. Hot repos are replicated for read traffic and warm caches in front.

How does code search work at GitHub scale?#

A separate index (trigram or sparse grep) ingests file content per push and serves queries with regex and language filters. The search service is fed by the same push pipeline that triggers webhooks.

How are pull requests implemented?#

A PR is metadata that points at two refs (base and head) plus review comments and merge state. The diff is computed on demand and cached; merging fast-forwards or creates a merge commit.

How does GitHub handle webhooks on push?#

Each push fans out into a webhook queue. Worker pods POST to registered URLs with HMAC signatures, retry on failure, and surface delivery logs in the UI.

Further reading#