Design GitHub Copilot#

Q: How does GitHub Copilot work under the hood?

Copilot is an IDE plugin that ships the cursor position, current file, and a ranked snippet of nearby code to a small specialised completion model over HTTPS. The model returns a short suggestion in under 100 milliseconds, which the IDE renders inline as ghost text. The user accepts with Tab or keeps typing to reject.

Q: Why is Copilot different from an agentic coding assistant?

Inline completion runs on every keystroke pause and must answer in under 100 milliseconds with a tiny ranked snippet of the repo. An agent runs for many seconds or minutes, makes tool calls, edits multiple files, and uses a frontier model with massive context. The two products share branding but have opposite latency and quality budgets.

Q: What context does Copilot actually send to the model?

It packs the current file around the cursor, recently edited files ranked by Jaccard similarity, open tabs, nearby symbols pulled from the language server, and a handful of repo level chunks retrieved by RAG. A 4k to 16k token prompt is typical for a 7B parameter model.

Q: What is a realistic Copilot acceptance rate?

Across all users and languages, accept rates hover near 25 to 35 percent. Strong languages like Python and TypeScript score higher; Bash and SQL score lower. The 95 percent figures you see in marketing decks are not real.

Q: How does a 20 dollar Copilot subscription cover ten thousand completions per month?

Each completion is roughly 100 input tokens and 30 output tokens on a 7B model running on shared inference hardware. Cost per completion is sub a tenth of a cent, so ten thousand completions cost about 10 cents in compute. Subscriptions cross subsidise heavy users with light ones.

Q: What stops Copilot from training on private code?

Enterprise tenants set a no train flag honored end to end, requests carry a data residency region tag, and prompts are dropped from telemetry pipelines after the accept reject signal is logged. Public tier completions can be retained for model improvement under the consumer terms.

A GitHub Copilot system design powers inline code completion: the moment you pause typing inside VS Code or JetBrains, the IDE ships your cursor position and a ranked snippet of repo context to a small specialised completion model that streams a suggestion back as ghost text in under 100 milliseconds. This is a fundamentally different product from an autonomous coding agent, even though both are sold under the Copilot brand.

Problem statement (interviewer prompt)

Design GitHub Copilot's inline completion system. The product lives inside an IDE, suggests the next 1 to 30 lines of code as the user types, and must return a useful completion in under 100 milliseconds end to end. Targets: 10 million daily active users, ~10k completions per user per month, sub 100 ms p50 time to first token, multi language and multi file context, accept reject telemetry, enterprise data residency, under 20 dollar per user per month all in cost.

flowchart LR
  IDE([IDE plugin<br/>VS Code, JetBrains])
  CTX[Context builder<br/>cursor, files, edits]
  EDGE[Edge gateway<br/>auth, region routing]
  MODEL[Completion model<br/>1.5B to 7B params]
  POST[Post-processor<br/>truncate, license filter]
  TEL[(Telemetry<br/>accept reject)]

  IDE --> CTX
  CTX --> EDGE
  EDGE --> MODEL
  MODEL --> POST
  POST --> IDE
  IDE -.signals.-> TEL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class IDE client;
    class EDGE edge;
    class CTX,POST service;
    class MODEL compute;
    class TEL datastore;

The latency budget is the whole story. A developer pauses for ~150 to 300 ms between bursts of typing. If the suggestion arrives after the next keystroke, the user has already moved on and the completion is wasted. So Copilot targets a sub 100 ms time to first token: roughly 30 ms for context building inside the IDE, 5 to 10 ms of network RTT to the nearest edge, 50 ms of inference on a small model, and ~10 ms of post-processing and render. Anything that adds 50 ms to that chain reduces accept rate by single digit percentage points, which compounds into a measurable revenue impact.

Context selection is the second story. A 7B model with a 16k token context cannot hold a million line repo. So a separate component decides what 4k to 8k tokens of nearby code best predict what the user is about to type: the current file around the cursor, the last 10 files the user edited, the contents of open tabs, and language server symbol lookups for the types referenced under the cursor. Repo level RAG sits on top of this for cross file completions where the right answer requires importing from a sibling module.

Acceptance rate is the only metric that matters. Did the user press Tab? Did they edit the suggestion before accepting it? Did they delete it within 30 seconds? These three signals, aggregated per language, per model version, per context strategy, drive every product decision at Copilot scale. Marketing claims about "95% acceptance" are not real. The honest number sits around 25 to 35 percent across users, languages, and codebases.

Why inline completion is not an agentic coding assistant#

Treating Copilot and an agent like Cursor's compose mode or Claude Code as the same product is the single most common interview mistake. They share a name and not much else.

Dimension	Inline completion (Copilot)	Agentic coding (Cursor compose, Claude Code)
Trigger	Keystroke pause, on every cursor move	Explicit user prompt
Latency budget	Sub 100 ms p50 TTFT	5 to 120 seconds total
Context size	4k to 16k tokens	200k to 1M tokens
Model class	1.5B to 7B specialised code model	Frontier model (GPT 4, Claude Sonnet)
Output length	1 to 30 lines, typically a few tokens	Multiple files, hundreds to thousands of lines
Tools	None during completion	File system, terminal, browser, MCP
Streaming UX	Ghost text drop in, no chunks	Mandatory; user watches output appear
Cost per call	< 0.1 cent	5 to 50 cents
Accept signal	Tab press within 5 seconds	Diff approval, run tests, commit
Quality bar	Useful guess often	Correct, runnable, tested code

This table is worth memorising. Every architectural decision in the rest of this page follows from the top three rows: an inline completion has 100 ms to do its job, fits a small prompt, and runs on a tiny model. Get those constraints wrong and the system is unbuildable.

The reason both products coexist under the Copilot brand is psychological, not technical. Inline completion is the always on muscle memory layer: it earns trust by being fast and frequently right enough to leave on. The agent is the deliberate, deeper layer that solves bigger problems on demand. Different teams, different infrastructure, different SKUs, same monthly invoice.

Top-level architecture#

flowchart TB
  subgraph IDE_Side[IDE side]
    PLUG([Copilot plugin])
    LSP[Language server<br/>symbol lookup]
    BUF[Open buffers<br/>edit history]
    CTX[Context builder<br/>rank + pack]
  end

  subgraph Edge[Edge tier]
    DNS[GeoDNS]
    ALB[Anycast LB]
    GW[Gateway<br/>auth, quota, telemetry]
    REG[Region router<br/>data residency]
  end

  subgraph Repo[Repo intelligence]
    INDEX[(Repo index<br/>per workspace)]
    EMB[Embedder<br/>code chunks]
    VEC[(Vector store<br/>chunk to embedding)]
    SYM[(Symbol graph<br/>references, defs)]
  end

  subgraph Inference[Inference fleet]
    PREFIX[Prefix cache<br/>KV by hash]
    SMALL[Small model<br/>1.5B latency tier]
    MID[Mid model<br/>7B quality tier]
    POST[Post-processor<br/>truncate, filter]
  end

  subgraph Safety[Safety + policy]
    LIC[License filter<br/>verbatim public code]
    SECRET[Secret detector<br/>API keys, tokens]
    POLICY[Policy engine<br/>no train, content filter]
  end

  subgraph Data[Telemetry plane]
    EVT[[Event bus<br/>Kafka]]
    AR[(Accept reject store<br/>ClickHouse)]
    BIL[(Billing rollups)]
    EVAL[(Model eval pipeline)]
  end

  PLUG --> LSP
  PLUG --> BUF
  LSP --> CTX
  BUF --> CTX
  CTX --> DNS
  DNS --> ALB
  ALB --> GW
  GW --> REG
  REG --> PREFIX
  PREFIX --> SMALL
  PREFIX --> MID
  SMALL --> POST
  MID --> POST
  POST --> LIC
  LIC --> SECRET
  SECRET --> PLUG
  CTX --> INDEX
  INDEX --> EMB
  EMB --> VEC
  INDEX --> SYM
  GW --> POLICY
  PLUG -.signals.-> EVT
  EVT --> AR
  EVT --> BIL
  AR --> EVAL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class PLUG client;
    class DNS,ALB,GW,REG edge;
    class CTX,LSP,BUF,POST,LIC,SECRET,POLICY service;
    class PREFIX cache;
    class INDEX,VEC,SYM,AR,BIL,EVAL datastore;
    class EVT queue;
    class EMB,SMALL,MID compute;

Three things stand out in this picture. First, the context builder lives inside the IDE, not on the server. Round trips for context assembly would blow the latency budget, and the language server is already in process anyway. Second, there are two model tiers: a sub 7B latency optimised model for the common path and a slightly larger quality model for harder completions, selected based on the request shape. Third, the safety layer runs server side after generation; a license filter that takes 30 ms is fine because it pipelines with the post-processor.

Sub-100 ms latency budget#

The entire reason Copilot has its particular shape is the 100 ms target. Walk through where the milliseconds go:

flowchart LR
  K[Keystroke pause<br/>200 ms idle trigger]
  C[Context build<br/>30 ms]
  N[Edge RTT<br/>5 to 10 ms]
  Q[Queue at inference<br/>2 ms]
  T[Prompt tokenize<br/>3 ms]
  P[Prefill<br/>10 ms KV cached]
  D[Decode tokens<br/>30 ms for 20 tokens]
  R[Network back<br/>5 ms]
  X[Post-process<br/>5 ms]
  G[Render ghost text<br/>5 ms]

  K --> C --> N --> Q --> T --> P --> D --> R --> X --> G

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    class K,C,X,G service;
    class N,R edge;
    class Q,T,P,D compute;

A few non obvious points hide in this picture:

The 200 ms idle trigger is not part of the budget; it is the threshold the IDE waits before deciding "the user actually paused, fire a request." Lowering it to 100 ms doubles the request rate and the cost without raising accept rate.
Streaming is optional. Most completions are short (under 60 tokens). Sending them as a single response after decode is simpler than SSE and saves the streaming protocol overhead. Long completions (function bodies) do stream.
Prefix caching is the single biggest server side optimisation. If the user typed one more character and resent the same prompt, the KV cache from the previous prefill is reused. Hit rate on prefix cache is around 60 to 80 percent during active editing.
Decode time dominates only for long completions. For a 5 token autocomplete, prefill dominates. This is why batch size and prefill throughput matter more than tokens per second for inline workloads.

If any one box blows its budget, the cascade hurts. A 200 ms model is unusable. A 50 ms model with 100 ms RTT is unusable. Copilot infrastructure has to be regional, with inference fleets in every major data residency zone, because trans Atlantic RTT alone eats half the budget.

Context builder pipeline#

The context builder is the IDE's secret weapon. It runs in milliseconds and decides what tiny slice of the codebase the model gets to see.

flowchart TB
  T[Trigger<br/>cursor pause]
  A[Current file<br/>500 lines around cursor]
  B[Open tabs<br/>top 5 by recency]
  C[Recent edits<br/>last 10 changed files]
  D[Language server<br/>symbols under cursor]
  E[Repo RAG<br/>vector top K]

  R[Ranker<br/>Jaccard + recency + symbol weight]
  P[Packer<br/>token budget aware]
  PROMPT[Prompt<br/>4k to 8k tokens]

  T --> A
  T --> B
  T --> C
  T --> D
  T --> E
  A --> R
  B --> R
  C --> R
  D --> R
  E --> R
  R --> P
  P --> PROMPT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class T,A,B,C,D,E,PROMPT service;
    class R,P compute;

What goes into the prompt#

The packer assigns each candidate snippet a score and fills the budget greedily. Categories, in rough priority order:

Current file local context (~30 to 40 percent of budget). The 200 lines before the cursor and 100 lines after. Imports go at the top, then the body. This is the highest signal source.
Recently edited files (~20 percent). Files the user touched in the last 10 minutes, ranked by Jaccard similarity of their identifier set with the current file. The intuition is simple: if you just edited userService.ts and now you are in userController.ts, the service file is highly relevant.
Open tabs (~15 percent). Tabs the user has open right now, even if not recently edited, signal mental working set.
Symbol definitions (~15 percent). For each unresolved symbol in the current file's nearby lines, the language server can give us its definition. A function call to parseUser benefits from seeing function parseUser(raw: string): User.
Repo RAG chunks (~10 percent). Optional. Useful for big repos where the user is calling into a part of the codebase they have not touched recently. Costs a vector lookup that adds 10 to 20 ms; many users disable it for that reason.

Jaccard ranking in practice#

The reason Jaccard similarity (intersection over union of identifier sets) works so well is that code shares vocabulary across related files. Two files about the same domain share type names, variable names, and import paths. Two files about unrelated domains do not. Computing Jaccard over a few hundred files in the project takes about 10 ms on a developer laptop and beats almost every more complex approach for this use case.

The packer must also handle the left right asymmetry of fill in the middle (FIM) models. Modern code completion models are trained with prefix and suffix tokens around the cursor, so the prompt format looks like <|fim_prefix|>before<|fim_suffix|>after<|fim_middle|>. The model fills the middle. This is critical: a "complete from cursor" prompt with no suffix performs much worse than a FIM prompt that includes the lines after the cursor.

Multi file and repo level context#

For repos beyond a few thousand files, the IDE side ranker is not enough. Cross file completions, like calling a function defined in a sibling module by its right argument order, need a richer view.

flowchart TB
  R[Repo on disk]
  CHUNK[Semantic chunker<br/>function and class units]
  EMB[Code embedder<br/>code specific model]
  VEC[(Vector store<br/>per workspace)]
  Q[Query at completion time]
  ANN[ANN search top 5]
  RANK[Cross encoder rerank<br/>optional]
  RES[Inject as repo context]

  R --> CHUNK
  CHUNK --> EMB
  EMB --> VEC
  Q --> ANN
  VEC --> ANN
  ANN --> RANK
  RANK --> RES

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    class R,Q,RES service;
    class CHUNK,EMB,ANN,RANK compute;
    class VEC datastore;

Some design constraints unique to this RAG path:

Indexing is incremental and local first. The full repo index lives on the developer's machine, not on the server, both for privacy and for latency. A background indexer watches the file system and re embeds changed chunks.
Chunks are syntactic, not fixed length. A function or class declaration is one chunk; chunking across a function boundary destroys the model's ability to reuse the result.
Embeddings come from a code specific model. General text embedders treat code as a bag of identifiers. Code embedders (CodeBERT, GTE Code, Voyage Code) are trained to map similar functions to similar vectors regardless of variable renaming, which matters a lot for cross repo retrieval.
Server side repo index is opt in. Enterprise tenants who want cross device parity (same suggestions on laptop and codespace) pay to host the index server side. Free tier users get a local only index.

The ANN search must finish in under 20 ms or it blows the budget. For most workspaces that means a 100k chunk index in memory with HNSW; only very large monorepos need disk backed structures.

Model selection and serving#

The third pillar is the model fleet. Inline completion does not use a frontier model; it uses a code specific model in the 1.5B to 7B range. There are a few candidates and a real reason to mix tiers.

Model family	Params	Approximate p50 latency on H100	Use case
Codex (original Copilot)	12B	~70 ms for 20 tokens	Original 2021 vintage; retired
StarCoder 2	3B / 7B / 15B	~30 to 80 ms	Open weight baseline
DeepSeek Coder	1.3B / 6.7B / 33B	~25 to 90 ms	Strong open weight, used for OSS Copilots
Qwen 2.5 Coder	1.5B / 7B / 32B	~25 to 80 ms	Recent and competitive
Codestral	22B	~120 ms	Mid quality, slower than the 7B class
GPT 4o mini	undisclosed	~150 ms	Used in newer Copilot quality tier

For the latency tier, a 1.5B to 7B model with 4 bit quantisation on a shared GPU pool serves 80 percent of completions. For longer or multi line completions, the gateway routes to a larger model (7B unquantised or a 22B class) with a slightly relaxed budget. The router decides on signals like: cursor at end of line vs inside a block, expected completion length, whether the user is in a heavy edit burst (where short is best) vs a deliberate pause (where quality matters).

Serving is continuous batching on top of a paged KV cache (vLLM, TensorRT LLM, or in house equivalents). Continuous batching lets the inference server interleave many in flight prompts in the same forward pass, which is the only way to hit thousand QPS per GPU with sub 100 ms p99. Prefix caching is the second optimisation: a developer typing one character at a time produces prompts that share 99 percent of their prefix with the previous one, and reusing the KV state for that prefix turns a 50 ms prefill into a 5 ms one.

Telemetry and accept rate#

The single most important signal in the whole system is did the user accept this completion. Everything else, every model version, every context strategy, every prompt template, is judged by accept rate aggregated across cohorts.

sequenceDiagram
  participant IDE as IDE plugin
  participant GW as Gateway
  participant T as Telemetry pipeline
  participant DB as Accept reject store

  IDE->>GW: completion request id=abc123
  GW-->>IDE: suggestion text + id
  IDE->>IDE: show ghost text
  alt User presses Tab
    IDE->>T: accept id=abc123 latency=85ms
  else User keeps typing
    IDE->>T: reject id=abc123 reason=typed-over
  else User accepts then deletes within 30s
    IDE->>T: accept-then-revert id=abc123
  end
  T->>DB: append event
  Note over DB: aggregated per language, model, context strategy

What we measure#

Accept rate: percent of suggestions accepted. Healthy: 25 to 35 percent across all users.
Edit distance after accept: how much did the user modify the accepted suggestion. Useful: more sensitive to subtle quality drops than accept alone.
Time to accept: distribution of how long after suggestion appears the user presses Tab. A flat distribution means users skim; a sharp spike near 1 second means users instantly know.
Persistence at 30 seconds, 5 minutes: did the accepted suggestion stay in the file or did the user delete and rewrite it. This is the most honest quality signal we have.
Per language slices: TypeScript and Python typically lead accept rates; shell scripts and SQL lag. Per language slicing exposes where the model is weakest.

The output of this pipeline feeds the model training and routing decisions. When a new model version goes live, it runs alongside the incumbent at 5 percent traffic for a week, with statistical tests comparing persistence rate. Only versions that beat incumbent at p < 0.01 across at least three language slices ship.

The "95 percent acceptance rate" figure shows up in vendor marketing and is misleading because it conflates "user accepted any suggestion offered today" with "user accepted this specific suggestion." On a per suggestion basis, 25 to 35 percent is the real number.

API shape and example code#

The IDE plugin sends a single POST to a regional gateway. The shape is mostly standard but with a few non obvious fields.

{
  "request_id": "abc123",
  "language": "typescript",
  "filename": "src/auth/login.ts",
  "prefix": "export async function login(email: string, password: string) {\n  const user = await db.users.findByEmail(email);\n  if (!user) return null;\n  ",
  "suffix": "\n  return generateToken(user);\n}",
  "open_files": [
    {"path": "src/db/users.ts", "content_hash": "sha256:..."},
    {"path": "src/auth/token.ts", "content_hash": "sha256:..."}
  ],
  "repo_chunks": [
    {"path": "src/db/users.ts", "snippet": "export const users = { findByEmail: ..."}
  ],
  "context_tokens": 2840,
  "max_tokens": 64,
  "model_hint": "latency",
  "tenant": {"id": "ent-acme", "no_train": true, "region": "eu-central"}
}

The response under happy path:

{
  "request_id": "abc123",
  "completion": "const ok = await bcrypt.compare(password, user.passwordHash);\n  if (!ok) return null;",
  "model": "code-7b-v3",
  "tokens": 24,
  "ttft_ms": 48,
  "total_ms": 78,
  "filtered": false,
  "cache_hit": "prefix"
}

A simplified context builder (the meat of the IDE side logic) in TypeScript:

type Candidate = { source: string; text: string; score: number };

function buildContext(
  currentFile: string,
  cursor: number,
  openTabs: string[],
  recentEdits: { path: string; content: string; editedAt: number }[],
  symbolLookups: { name: string; defText: string }[],
  budgetTokens: number,
): { prefix: string; suffix: string; extras: Candidate[] } {
  const prefix = currentFile.slice(Math.max(0, cursor - 4_000), cursor);
  const suffix = currentFile.slice(cursor, cursor + 1_500);

  const localIds = new Set(extractIdentifiers(currentFile));
  const candidates: Candidate[] = [];

  for (const f of recentEdits) {
    const ids = new Set(extractIdentifiers(f.content));
    const jaccard = intersect(localIds, ids).size / union(localIds, ids).size;
    const recency = Math.exp(-(Date.now() - f.editedAt) / (5 * 60_000));
    candidates.push({
      source: f.path,
      text: f.content.slice(0, 1_200),
      score: 0.6 * jaccard + 0.4 * recency,
    });
  }

  for (const sym of symbolLookups) {
    candidates.push({
      source: `sym:${sym.name}`,
      text: sym.defText,
      score: 0.9,
    });
  }

  candidates.sort((a, b) => b.score - a.score);

  let used = approxTokens(prefix) + approxTokens(suffix);
  const extras: Candidate[] = [];
  for (const c of candidates) {
    const cost = approxTokens(c.text);
    if (used + cost > budgetTokens) continue;
    extras.push(c);
    used += cost;
  }
  return { prefix, suffix, extras };
}

The real production version is more complex (it dedupes across sources, drops noisy comments, normalises whitespace before Jaccard, applies a content based hash for prefix cache friendliness, and includes a fast circuit breaker that drops the heaviest candidates first when the budget is tight) but the skeleton above captures 80 percent of what matters.

Privacy, data residency, and the "no train" promise#

A surprising amount of Copilot architecture exists to make enterprises comfortable enough to install it.

No train flag. Enterprise tenants set a flag that flows with every request. The telemetry pipeline strips prompt and completion content from accepted suggestions; only the metadata (accept reject, language, latency) is retained for billing and model evaluation. Public tier users have a different toggle that defaults to on for model improvement.
Data residency. Every request carries a region tag; the gateway routes to inference fleets in the same region (EU, US, Australia). For sovereign deployments, the model serves entirely inside the customer's own cloud region.
TLS everywhere. IDE to edge is TLS 1.3 with certificate pinning in the plugin. Edge to inference is mTLS inside a private network.
Secret scanning. A pre filter on the prompt drops any chunk that looks like an API key, JWT, SSH key, or credit card. This is cheap insurance against accidentally training on, or leaking, secrets that crept into source.
License filter on output. Public models can occasionally regurgitate verbatim training data. A post filter checks the output against a hash of known public code (subset of GitHub public repos) and suppresses any suggestion that is a near exact match longer than ~150 tokens.
Audit log. Per request metadata (who, when, model, tokens, accepted) is retained per the tenant's compliance setting. Body content is not.

The combined effect is that an enterprise security review can verify, end to end, that prompts and completions never leave their region, never enter a training set, and never persist beyond the audit window.

Cost economics: how does 20 dollars per month work#

A Copilot consumer subscription costs around 20 dollars per month and a user generates roughly 10 thousand completion requests in that month, of which maybe 30 percent are accepted. Each completion runs through a small model on shared inference hardware.

Back of envelope:

Average input: ~3000 tokens packed prompt.
Average output: ~25 tokens (most completions are short).
7B model at int4, decode at ~150 tokens per second per GPU, with continuous batching the effective per request cost is roughly 0.3 GPU milliseconds for prefill and 0.2 GPU milliseconds for decode.
H100 hour: ~3 dollars on a cloud, lower at hyperscaler scale.
Per completion compute cost: order of 0.04 cents at decent batch fill, plus ~0.01 cents for context and post processing.
10k completions per user per month: ~5 cents in raw compute.

Add roughly 4x for storage (prefix cache, prompt cache, telemetry storage), networking, the language server / RAG infrastructure, and engineering overhead. Realistic landed cost per active user per month is in the range of 3 to 8 dollars. A 19 dollar subscription leaves comfortable margin for the long tail of power users who burn 5x the average and for the free trial conversion funnel.

Compare to agent SKUs (Cursor Pro, Copilot Workspace) at 40 to 60 dollars per month where a single agent run can cost a dollar. Inline completion is the low margin volume product; the agent is the high margin per session one. Most vendors bundle both into a single subscription so the cheap inline product subsidises the expensive agent runs and vice versa.

Compare to: Cursor, Codeium, Tabnine, Replit, Amazon Q#

Anyone designing this system gets asked about competitors. The differences are subtler than marketing implies.

Product	Completion model	Repo context	Agent mode	Pricing	Notable choice
GitHub Copilot	Mix: GPT class for chat, custom code models for inline	Local LSP + optional server side index	Yes (Copilot Workspace)	19 dollars personal, 39 dollars enterprise	Default for VS Code, deep IDE integration
Cursor	Inline: custom small model; agent: Claude / GPT 4 class	Aggressive local + server side embedding	Yes (compose)	20 dollars Pro, 40 dollars Business	Forked VS Code, agent is the headline product
Codeium / Windsurf	Self hosted small model on user's region	Local index	Yes (Cascade)	Free tier, 15 dollars enterprise	Strong privacy story, free generous tier
Tabnine	Smaller local model (laptop CPU inference)	Local repo	Limited	12 to 39 dollars	Air gapped option, slower but private
Replit Ghostwriter	Custom replit code model	Replit project only	Yes (Agent)	Bundled with Replit	Tied to Replit IDE, less generic
Amazon Q Developer	Bedrock backed code model	AWS aware	Yes	Free personal, 19 dollars Pro	AWS service knowledge, weaker base

Two themes run through this list. First, every serious player splits inline from agent: same product family, separate infra. Second, the inline completion quality has converged. Differentiation is now in IDE depth (Copilot in VS Code is hard to beat), agent capability (Cursor leads on developer experience), or privacy posture (Codeium and Tabnine win the locked down enterprise).

Failure modes and scaling#

Failure	Effect	Mitigation
Inference fleet saturated in one region	Latency spikes	Cross region overflow with degraded quality flag
Model returns gibberish completion	Accept rate drop, user trust loss	Canary deploys, automated rollback on persistence drop
Context builder ranks wrong files	Quality drop on multi file repos	A B test ranking changes per language slice
Repo RAG index grows unbounded	Memory pressure on user laptop	Hard cap per workspace, LRU eviction
License filter false positive	Suggestion suppressed silently	Telemetry flag, threshold tuning by language
User keystroke storm	Wasted requests, cost spike	IDE side debounce, dedupe in flight prompts on same prefix
Telemetry pipeline backed up	Stale accept rate dashboards	Backfill from accept logs, dual write to ClickHouse + Kafka
Enterprise no train flag dropped	Compliance violation	Flag baked into request signature, verified at telemetry sink

Scaling: gateway and inference are horizontally scaled per region. Hot region (US East) at peak serves on the order of tens of thousands of completions per second. KV prefix cache is the highest leverage per GPU optimisation; doubling cache memory often beats adding 20 percent more GPUs. The repo index is the only stateful per user component; for the server side variant it lives in a per workspace partition of an OpenSearch or pgvector cluster.

Real numbers worth knowing#

These numbers come up in interviews. Memorise them.

~30 percent of newly written code at GitHub instrumented users is now Copilot suggested (accepted). Public figure as of 2024 to 2025.
Per suggestion accept rate: 25 to 35 percent across users and languages.
p50 time to first token: ~50 to 80 ms inside the data centre serving the user.
Average input tokens per request: 2 to 4 thousand.
Average output tokens accepted: ~20 to 30.
7B class model serves 80 percent of requests; larger model for the quality tier serves ~20 percent.
Repo RAG hit rate (when enabled): 40 to 60 percent of cross file completions use a retrieved chunk.
Active users globally: over 1.5 million paid seats by late 2024.

FAQ#

How does GitHub Copilot work under the hood?#

Copilot is an IDE plugin that ships the cursor position, current file, and a ranked snippet of nearby code to a small specialised completion model over HTTPS. The model returns a short suggestion in under 100 milliseconds, which the IDE renders inline as ghost text. The user accepts with Tab or keeps typing to reject.

Why is Copilot different from an agentic coding assistant?#