Coding Agent

A coding agent system design powers products like Cursor, GitHub Copilot, and Claude Code. The agent reads files in your repo, edits them, runs shell commands, and converses with you, all while staying within a tight latency and cost budget. Building one is mostly about gluing an LLM to a sandboxed workspace and a smart context retriever.

Problem statement (interviewer prompt)

Design an AI coding agent that integrates with an IDE, understands a million-line repo, can read and edit multiple files, runs terminal commands, and answers conversational follow-ups. Target sub-second first-token latency, 5 to 20 second task completion, and keep average per-task LLM spend under 10 cents.

flowchart LR
  IDE([IDE / Editor<br/>VS Code, JetBrains])
  CLI([Terminal client])
  GW[Agent Gateway<br/>auth, rate limit]
  ORCH[Agent Orchestrator<br/>ReAct loop]
  CTX[Context Retriever<br/>repo index + RAG]
  TOOL[Tool Runtime<br/>read_file, edit_file, bash]
  ROUTE[Model Router<br/>small / big]
  LLM[(LLM providers<br/>Anthropic, OpenAI)]
  IDX[(Repo Vector Index<br/>per-workspace)]
  MEM[(Conversation Memory<br/>Redis + Postgres)]
  SBX[Sandbox<br/>ephemeral container]

  IDE --> GW
  CLI --> GW
  GW --> ORCH
  ORCH --> CTX
  CTX --> IDX
  ORCH --> ROUTE
  ROUTE --> LLM
  ORCH --> TOOL
  TOOL --> SBX
  ORCH --> MEM

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class IDE,CLI client;
    class GW edge;
    class ORCH,CTX,ROUTE,TOOL service;
    class IDX,MEM datastore;
    class SBX compute;
    class LLM external;

Context is the hardest problem. A million-line repo will not fit in any context window, so the agent retrieves only the files and symbols that matter. The retriever blends three signals: semantic similarity from a code embedding index, a symbol graph that traces definitions and callers, and recently viewed or edited files from the IDE session. Bad retrieval poisons the prompt and the model hallucinates APIs that do not exist.

Tool use drives correctness. The model does not produce a final diff in one shot. It calls read_file, scrolls through results, calls grep, then proposes an edit and runs tests. Each tool call is sandboxed: the runtime executes inside an ephemeral container with read-only mounts plus a writable scratch directory, and shell commands are killed if they exceed a CPU or wall-clock budget. Diffs are presented to the user for approval before they touch the real working copy.

Cost and latency force model routing. Sending every keystroke completion to a frontier model would cost dollars per active user per day, so a small model handles autocomplete and trivial edits while the big model handles multi-file refactors and architectural questions. A separate prompt cache keyed by the file content and conversation prefix slashes repeated cost, since users typically iterate on the same files for hours at a time.

Problem statement (interviewer prompt)

Design a coding agent that competes with Cursor and Claude Code. It must integrate with IDEs and a CLI, understand a million-line monorepo, edit multiple files atomically, run terminal commands inside a sandbox, and answer follow-up questions. Targets: first-token p50 under 400 ms, multi-step task p50 under 12 s, 99.5% availability, and a per-task LLM cost under $0.10 averaged across 200k daily active developers.

Requirements

Functional

• Inline completion (Tab-style) and chat (multi-turn).
• Multi-file edit with diff preview and explicit user approval.
• Tools: read_file, edit_file, grep, glob, run_terminal, apply_patch.
• Awareness of open files, cursor location, selection, and recent edits.
• Continuity across sessions: previous conversations and outstanding TODOs persist.
• Per-workspace repo indexing with incremental updates on file save.

Non-functional

• First-token latency p50 under 400 ms, p99 under 1 s.
• Multi-step task p50 12 s, p99 30 s.
• 99.5% availability; degrade gracefully when one provider is down.
• 200k DAU, 4M agent invocations/day, peak 200 req/s.
• Per-task LLM cost under $0.10 averaged across small and big model traffic.
• No source code leaves the customer's VPC for enterprise tier.

Top-level architecture

flowchart TB
  subgraph Client[Client surfaces]
    VSC([VS Code extension])
    JET([JetBrains plugin])
    CLI([CLI / TUI])
    WEB([Web editor])
  end

  subgraph Edge[Edge tier]
    LB[L7 LB / Anycast]
    GW[Agent Gateway<br/>auth, rate limit, tenancy]
    WS[WebSocket fanout<br/>streaming tokens]
  end

  subgraph Core[Agent core]
    ORCH[Orchestrator<br/>ReAct loop, step budget]
    PLAN[Planner<br/>task decomposition]
    ROUTE[Model Router<br/>size + provider]
    CACHE[Prompt Cache<br/>prefix + file hash]
  end

  subgraph Context[Context layer]
    RET[Retriever<br/>hybrid: BM25 + vectors + symbols]
    IDX[(Vector Index<br/>code embeddings)]
    SYM[(Symbol graph<br/>tree-sitter + LSP)]
    SES[(Session state<br/>open files, cursor)]
  end

  subgraph Tools[Tool runtime]
    TOOL[Tool dispatcher]
    SBX[Container pool<br/>firecracker / gVisor]
    FS[Workspace FS<br/>overlay, read-only base]
    PATCH[Patch applier<br/>3-way merge]
  end

  subgraph Data[State and memory]
    CONV[(Conversations<br/>Postgres)]
    BLOB[(Blob store<br/>large outputs, diffs)]
    MEM[(Redis<br/>active session, tokens used)]
    USE[(Usage ledger<br/>ClickHouse)]
  end

  subgraph LLMs[Model providers]
    SMALL[Small model<br/>Haiku / 4o-mini]
    BIG[Big model<br/>Sonnet / GPT-4 / Opus]
    LOCAL[Self-hosted<br/>code model for completion]
  end

  VSC --> LB
  JET --> LB
  CLI --> LB
  WEB --> LB
  LB --> GW
  GW --> WS
  WS --> ORCH
  ORCH --> PLAN
  ORCH --> RET
  RET --> IDX
  RET --> SYM
  RET --> SES
  ORCH --> ROUTE
  ROUTE --> CACHE
  ROUTE --> SMALL
  ROUTE --> BIG
  ROUTE --> LOCAL
  ORCH --> TOOL
  TOOL --> SBX
  SBX --> FS
  TOOL --> PATCH
  ORCH --> CONV
  ORCH --> MEM
  ORCH --> BLOB
  ORCH -.usage events.-> USE

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class VSC,JET,CLI,WEB client;
    class LB,GW,WS edge;
    class ORCH,PLAN,ROUTE,RET,TOOL,PATCH service;
    class IDX,SYM,SES,CONV,USE datastore;
    class CACHE,MEM cache;
    class SBX,FS compute;
    class BLOB storage;
    class SMALL,BIG,LOCAL external;

Data model

-- Conversations and messages live in Postgres, sharded by workspace_id.
conversations(
  id uuid PK,
  workspace_id uuid,
  user_id uuid,
  title text,
  created_at timestamptz,
  last_activity_at timestamptz
);

messages(
  id uuid PK,
  conversation_id uuid FK,
  role text,            -- user | assistant | tool
  content_blob_id uuid, -- large blobs offloaded to S3
  tool_call jsonb,      -- name, args, result_summary
  tokens_in int,
  tokens_out int,
  model text,
  created_at timestamptz
);

-- Repo index metadata, with raw vectors in a separate vector DB.
repo_index(
  workspace_id uuid PK,
  repo_hash text,
  embedding_model text,
  total_chunks int,
  last_full_reindex timestamptz,
  bytes_indexed bigint
);

file_chunk(
  id uuid PK,
  workspace_id uuid,
  path text,
  start_line int,
  end_line int,
  symbol text,           -- enclosing function/class
  language text,
  content_hash text,
  vector_id text         -- foreign key into vector DB
);

-- Usage ledger fans into ClickHouse for billing rollups.
usage_event(
  ts DateTime64,
  workspace_id String,
  user_id String,
  model String,
  tokens_in UInt32,
  tokens_out UInt32,
  task_id String,
  cost_micros UInt64
);

Component deep-dive

Repo retriever

The repo retriever is the difference between a useful agent and a hallucinating one. It must answer "for this user question, which 8 KB of code should sit in the LLM's prompt?"

flowchart LR
  Q[User query + context]
  EXP[Query expansion<br/>small model]
  BM25[BM25 over file paths and symbols]
  VEC[Vector search<br/>code embeddings]
  GRAPH[Symbol graph walk<br/>callers, callees]
  RECENT[Recent files<br/>IDE session]
  FUSE[Reciprocal Rank Fusion]
  RERANK[Cross-encoder rerank<br/>top 50 to top 8]
  CTX[Final context bundle]

  Q --> EXP
  EXP --> BM25
  EXP --> VEC
  Q --> GRAPH
  Q --> RECENT
  BM25 --> FUSE
  VEC --> FUSE
  GRAPH --> FUSE
  RECENT --> FUSE
  FUSE --> RERANK
  RERANK --> CTX

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class Q,CTX service;
    class BM25,VEC,GRAPH,RECENT,FUSE,RERANK compute;
    class EXP service;

Chunking is symbol-aware: tree-sitter splits files at function and class boundaries instead of fixed token windows, which keeps complete definitions together. The vector index is hosted per-workspace because cross-workspace search is both unnecessary and a data-leak risk. Reindexing on file save is incremental: the watcher diffs content hashes, re-embeds only changed chunks, and updates the symbol graph by reparsing the affected files.

Tool runtime and sandbox

Tools run in ephemeral microVMs (Firecracker) or gVisor sandboxes. Each session gets one workspace container that mounts the user's working copy as an overlay filesystem: writes land in a tmpfs upper layer until the user explicitly accepts a diff.

sequenceDiagram
  participant O as Orchestrator
  participant T as Tool dispatcher
  participant S as Sandbox VM
  participant P as Patch applier
  participant U as User (IDE)

  O->>T: tool_call(edit_file, args)
  T->>S: apply edit in overlay FS
  S-->>T: success + diff
  T-->>O: tool_result
  O->>T: tool_call(run_terminal, "pytest")
  T->>S: exec with 60s timeout
  S-->>T: exit_code, stdout (truncated)
  T-->>O: tool_result
  O->>U: stream "ready, here's the diff"
  U->>P: accept diff
  P->>U: write to real working copy

The sandbox enforces three limits per call: CPU seconds, wall-clock seconds, and network egress. Network defaults to deny, with an allowlist for pip, npm, and the company's internal registry. Output streams are byte-capped: a runaway test suite cannot blow up the conversation context by returning 5 MB of logs, so the tool returns a head, tail, and a summary blob ID instead.

Model router

The router decides which provider and which model size handles each call.

flowchart TB
  IN[Incoming request<br/>type + tokens + tenant]
  CLASS{Task class}
  COMP[Autocomplete]
  CHAT[Chat / multi-file]
  TOOL[Tool-using step]
  RANK[Cost-aware ranking<br/>health + latency + price]
  POOL[Provider pool]
  FB[Fallback chain]
  OUT[Stream response]

  IN --> CLASS
  CLASS -->|completion| COMP
  CLASS -->|conversation| CHAT
  CLASS -->|tool step| TOOL
  COMP --> POOL
  CHAT --> RANK
  TOOL --> RANK
  RANK --> POOL
  POOL --> FB
  FB --> OUT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class IN,CLASS,COMP,CHAT,TOOL,RANK service;
    class POOL,FB compute;
    class OUT external;

Autocomplete uses a self-hosted small code model (StarCoder, DeepSeek-Coder) for sub-100 ms latency and predictable cost. Chat and tool steps go to a frontier model. The router maintains rolling health signals for every provider region: if Anthropic us-east-1 is degraded, it ships to OpenAI as the fallback for that tenant. Sticky routing within a single task pins all steps to one provider so the prompt cache stays warm.

Conversation memory

Memory has three layers with different lifetimes:

1. Active window (Redis): tokens of the current turn plus tool outputs, evicted after the task ends.
2. Conversation log (Postgres + S3 for large blobs): full message history, used to reconstruct context when the user returns.
3. Long-term workspace memory (vector index + curated notes): facts the agent learned about the codebase, like "uses pnpm workspaces" or "tests live in apps/web/__tests__/".

When a conversation crosses the model's context window, an async summarizer condenses older turns into a recap and the orchestrator continues from system + recap + last N turns. Idempotency keys on the tool call layer prevent double-applying an edit if the IDE retries.

Sample code: the tool dispatcher

import asyncio, time, uuid
from typing import Any

class ToolDispatcher:
    def __init__(self, sandbox, blob_store, max_output_bytes=64_000):
        self.sandbox = sandbox
        self.blob_store = blob_store
        self.max_output_bytes = max_output_bytes

    async def call(self, name: str, args: dict, task_id: str,
                   idempotency_key: str) -> dict[str, Any]:
        cached = await self.sandbox.get_idempotent(idempotency_key)
        if cached:
            return cached

        start = time.monotonic()
        try:
            if name == "read_file":
                out = await self.sandbox.read_file(args["path"], args.get("range"))
            elif name == "edit_file":
                out = await self.sandbox.apply_edit(
                    args["path"], args["old"], args["new"]
                )
            elif name == "run_terminal":
                out = await asyncio.wait_for(
                    self.sandbox.exec(args["cmd"], cwd=args.get("cwd")),
                    timeout=args.get("timeout_s", 60),
                )
            else:
                raise ValueError(f"unknown tool {name}")
        except asyncio.TimeoutError:
            return {"error": "timeout", "elapsed_ms": int((time.monotonic()-start)*1000)}

        if len(out.get("stdout", b"")) > self.max_output_bytes:
            blob_id = await self.blob_store.put(out["stdout"])
            out = {
                "stdout_head": out["stdout"][:8_000].decode(errors="replace"),
                "stdout_tail": out["stdout"][-8_000:].decode(errors="replace"),
                "stdout_blob_id": blob_id,
                "bytes_total": len(out["stdout"]),
                "exit_code": out["exit_code"],
            }
        out["elapsed_ms"] = int((time.monotonic() - start) * 1000)
        await self.sandbox.set_idempotent(idempotency_key, out, ttl_s=600)
        return out

Failure modes and scaling

Failure	Effect	Mitigation
LLM provider regional outage	Stream stalls mid-task	Per-tenant fallback chain in router, replay last step on new provider
Sandbox VM hangs	One user task stuck	Wall-clock budget kills VM; orchestrator retries with --no-network
Retriever returns stale chunks after rename	Wrong context, broken edits	File watcher reindexes on save, symbol graph validated against tree-sitter
Hot tenant overloads vector DB	Latency tail spikes	Per-workspace shard, token bucket at gateway
Runaway agent loop	Cost spike, no progress	Step budget (default 25), no-progress detector, hard token cap
Memory store loss	Lost in-flight conversation	Conversation log is the source of truth, Redis is rebuildable

Scale-out is per-workspace. The orchestrator is stateless: any pod can handle any request because state lives in Redis and Postgres. The vector index is partitioned by workspace_id; large enterprise tenants get dedicated index shards on bigger instances. Sandbox VMs warm-pool by language: a pool of pre-booted Python and Node containers keeps cold-start under 200 ms.

Cost considerations

Assume 200k DAU, 20 tasks/user/day, 4M tasks/day. Token mix per task: 8k input + 2k output average, 70% on the big model, 30% on the small completion model.

• Big model: 4M tasks/day * 0.7 = 2.8M tasks * (8k input * $3/1M + 2k output * $15/1M) = 2.8M * ($0.024 + $0.030) = $151k/day raw.
• Prompt caching cuts repeated context by ~70% on chat workloads: effective spend ~$50k/day.
• Small completion model self-hosted on 100 H100s at ~$30/hour each = $72k/day fixed, serving billions of completions/day at near-zero marginal cost.
• Sandbox VMs: 200 concurrent peak * $0.05/hour * 24 = ~$240/day.
• Vector DB: 200k workspaces * 50 MB avg = 10 TB, ~$5k/month managed.

Per-task cost ends at roughly $0.04 with caching, well inside the $0.10 budget. The biggest knobs are prompt cache hit rate and routing: a 10-point cache improvement saves more than any infrastructure optimization.

Latency budget

Target: first-token p50 under 400 ms.

Step	Budget	Notes
Network IDE to gateway	30 ms	TLS keep-alive, regional POPs
Auth + rate limit	10 ms	JWT verify, Redis token bucket
Context retrieval	120 ms	Vector + BM25 parallel, rerank 50 to 8
Prompt assembly	20 ms	Template render, cache key compute
Provider TTFT	200 ms	Sonnet streaming, cache hit ~80%
Edge to client	20 ms	WebSocket streaming
Total p50	400 ms

For multi-step tasks, add tool-call round trips: each read_file is roughly 80 ms (sandbox + LLM continuation), run_terminal runs as long as the command takes. A typical 4-step task at 12 s is mostly model inference time, not infrastructure.

Quick reference

Functional requirements

• Inline completion + multi-turn chat.
• Multi-file edits with diff approval.
• Tools: read_file, edit_file, grep, glob, run_terminal, apply_patch.
• Session-aware: open files, cursor, recent edits.
• Persistent conversations across IDE restarts.
• Per-workspace repo indexing with incremental updates.

Non-functional

• First-token p50 under 400 ms, p99 under 1 s.
• Multi-step task p50 under 12 s.
• 99.5% availability with provider fallback.
• Per-task LLM cost under $0.10.

Capacity (back-of-envelope)

• 200k DAU, 20 tasks/user/day, 4M tasks/day.
• Peak 200 req/s, 8k input + 2k output tokens/task avg.
• 200k workspaces * 50 MB vector index = 10 TB vector DB.
• Big model spend ~$50k/day with prompt caching.
• Self-hosted small model: ~100 H100s for completions.

Architecture pillars

• Gateway: auth, tenant routing, rate limit, WebSocket streaming.
• Orchestrator: stateless ReAct loop with step budget (default 25).
• Retriever: hybrid BM25 + vectors + symbol graph + recent files, RRF + cross-encoder rerank.
• Tool runtime: ephemeral microVMs (Firecracker / gVisor), overlay FS, network-deny default.
• Model router: small for autocomplete, big for chat, sticky within a task, per-tenant fallback chain.
• Memory: Redis (active), Postgres + S3 (logs), vector index (workspace facts).

Model routing rules

Task	Model	Why
Inline completion	Self-hosted small code model	Sub-100ms, cost-stable
Quick chat / explain	Mid-tier (Haiku, 4o-mini)	Good enough, cheap
Multi-file refactor	Frontier (Sonnet / GPT-4 / Opus)	Reasoning quality
Tool-using step	Frontier with prompt cache	Cache hit rate >70%
Fallback	Cross-provider	Survive single-provider outage

Tool runtime checklist

• Wall-clock + CPU budgets per call.
• Network deny by default, allowlist for package managers.
• Output byte cap, head + tail + blob_id pattern.
• Idempotency keys to dedupe IDE retries.
• Overlay FS, real working copy only on user accept.
• Pre-warmed VM pool per language for sub-200ms cold start.

Common pitfalls

• Embedding the whole repo every save instead of incremental reindex by content hash.
• Sending the full file on every completion instead of cursor-window + symbol context.
• Allowing the LLM to write directly to disk (must go through diff approval).
• No step budget, agent loops forever on ambiguous tasks.
• Forgetting prompt cache: same conversation reprices on every turn.
• Cross-tenant retriever, leaking workspace data.
• Ignoring tool output truncation, runaway test logs blow context.

Refs

• Cursor engineering blog, GitHub Copilot deep dives, Anthropic Claude Code architecture posts, Sourcegraph Cody papers, Aider chat repo, OpenDevin / SWE-Bench writeups.

FAQ

How does a coding agent understand a large repo?

It indexes the workspace (symbol search, embeddings over chunks, and recent-file heuristics) and packs only relevant snippets into the prompt, instead of feeding the full repo to the LLM.

How is sub-second first-token latency achieved?

Stream from the LLM provider, prefetch likely context as the user types, and use smaller models or speculative decoding for low-latency replies, escalating to larger models for hard tasks.

How does the agent safely edit multiple files?

It proposes diffs as structured tool calls, applies them in a sandbox or worktree, and gates writes behind user approval or a test run so a bad edit never silently lands on disk.

How is per-task cost kept low?

Model routing sends simple tasks to cheap models (Haiku, GPT-4o-mini) and escalates complex ones; prompt caching reuses repo context across turns to cut input tokens by 70 to 90 percent.

What tools does a coding agent typically expose?

Read file, write file, run shell, search codebase, run tests, and apply diff. Each tool has a JSON schema so the LLM's structured output can be validated and dispatched safely.

Related Topics

• Tool Use and Function Calling: tool schemas and execution patterns
• Agent Loop and ReAct: the perceive-think-act cycle the orchestrator runs
• Agent Memory: conversation, working, and long-term memory tiers

Video walkthrough

How Coding Agents Actually Work (Animated) : via System Design Tutorial