Skip to content

Design Perplexity (AI Search)#

A Perplexity-style AI search system design turns a free-form question into a grounded answer with inline citations, by combining a query planner, hybrid web retrieval, cross-encoder reranking, and a synthesizer LLM that refuses claims no retrieved source supports. Perplexity, ChatGPT browse, You.com, and Google AI Overviews all live in this niche; the architecture below is the canonical reference for an AI search engine interview.

Problem statement (interviewer prompt)

Design an AI search engine in the style of Perplexity. Given a natural language question, return a concise grounded answer with inline numbered citations linking to live web sources, plus three follow-up question suggestions. Targets: p50 first token in 1.5 seconds, p95 full answer in 6 seconds, 100k queries per minute peak, 95% citation precision (every cited URL actually supports the cited claim).

flowchart LR
  U([User query])
  PLAN[Query planner<br/>intent + rewrite]
  FAN[Retrieval fanout]
  WEB[Web index<br/>own crawl or BSP]
  VEC[Vector cache<br/>cached docs]
  RT[Real-time fetcher<br/>fresh URLs]
  RR[Reranker<br/>cross-encoder]
  READ[Reader<br/>passage extract]
  SYN[Synthesizer<br/>grounded answer]
  CHK[Citation check]
  ANS([Answer with refs])
  FU[Follow-up gen]

  U --> PLAN
  PLAN --> FAN
  FAN --> WEB
  FAN --> VEC
  FAN --> RT
  WEB --> RR
  VEC --> RR
  RT --> RR
  RR --> READ
  READ --> SYN
  SYN --> CHK
  CHK --> ANS
  ANS --> FU

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    class U,ANS client;
    class PLAN,FAN,SYN,FU service;
    class RR,READ,CHK compute;
    class WEB,VEC datastore;
    class RT external;

The planner is the brain. It reads the raw query, classifies intent (factual lookup, news, comparison, how-to, navigational), rewrites the phrasing into a search-engine-friendly form, and decomposes compound questions into independent sub-queries that can be retrieved in parallel. "Who won the 2024 election and what was the margin" becomes two sub-queries with one merged answer.

Retrieval is hybrid and parallel. The system never trusts a single source. It hits a classical web index (own crawl or a wholesale partner like Brave / Bing / Tavily), a vector store of recently cached documents for semantic recall, and a real-time HTTP fetcher for fresh URLs that the index has not yet seen. All three fanout in parallel; the slowest path bounds latency.

The synthesizer composes with refusal. A small reader model extracts the most relevant passages from the top reranked pages and attributes each to its source URL. The synthesizer then writes the final answer, citing each claim with a numbered marker like [1], [2], and explicitly refusing claims that no retrieved passage supports. A citation coverage check after generation enforces grounding.

Requirements#

Functional

  • Accept natural language questions; no structured query language.
  • Classify intent (factual, news, comparison, how-to, navigational, code).
  • Rewrite and decompose compound queries.
  • Retrieve from a web index, a vector cache, and a real-time fetcher.
  • Rerank candidate documents with a cross-encoder before reading.
  • Stream the answer token by token while later citations resolve.
  • Render inline numbered citations linking to source URLs.
  • Generate 3 to 5 follow-up suggestions per answer.
  • Support multi-turn conversations that reuse prior retrieved evidence.
  • Refuse to answer when no source supports the claim.

Non-functional

  • p50 first token in 1.5 seconds, p95 full answer in 6 seconds.
  • 100k queries per minute peak, 5k QPS sustained.
  • 95% citation precision: cited URLs actually support cited claims.
  • 99.9% availability of the answer endpoint.
  • Freshness: news queries must surface content under 5 minutes old.
  • Cost target under $0.01 per query at scale.

What makes AI search distinct#

A classic web search engine ranks documents and shows ten blue links. A naive RAG system retrieves from a private vector store and writes a grounded answer over that. AI search sits in between and is harder than either.

It must rank documents (like search), retrieve and ground (like RAG), but also handle the live public web, attribute every sentence to a URL, generate follow-ups, refuse when evidence is weak, and stream all of this in under two seconds. The combination is what makes the system non-trivial.

Property Classic web search Naive RAG AI search (Perplexity)
Source Public web Private corpus Public web + cached pages
Output Ranked links Free-form answer Grounded answer + links
Citations Implicit (the SERP) Optional Mandatory, inline numbered
Freshness Crawl-bounded Static corpus Real-time fetch on demand
Refusal Not applicable Rare Required on weak evidence
Follow-ups Suggestions Not present Generated per answer
Latency target 200 ms 2 to 5 seconds 1.5 s first token, 6 s done

Top-level architecture#

flowchart TB
  subgraph Client[Clients]
    UI([Web UI])
    APP([Mobile app])
    API([API consumer])
  end

  subgraph Edge[Edge]
    LB[Anycast LB]
    GW[Gateway<br/>auth, rate limit]
    SSE[SSE stream]
  end

  subgraph Plan[Planning tier]
    INT[Intent classifier<br/>small model]
    RW[Query rewriter]
    DEC[Decomposer]
    QC[(Query cache<br/>normalized key)]
  end

  subgraph Retr[Retrieval tier]
    WEB[Web index<br/>own or BSP API]
    VEC[(Vector store<br/>cached docs)]
    RT[Real-time fetcher<br/>HTTP + reader]
    MERGE[Result merger]
  end

  subgraph Rank[Ranking tier]
    BM25[BM25 score]
    EMB[Embed top N]
    XENC[Cross-encoder<br/>rerank]
    DEDUP[Dedup by domain]
  end

  subgraph Synth[Synthesis tier]
    READ[Reader<br/>passage extract]
    SYN[Synthesizer<br/>frontier model]
    COV[Citation coverage]
    FU[Follow-up gen]
  end

  subgraph Data[Data plane]
    CRAWL[(Web crawl + index)]
    DOCS[(Doc store<br/>cleaned pages)]
    ANS[(Answer log<br/>session memory)]
    EVT[[Kafka events]]
    OBS[Metrics + traces]
  end

  UI --> LB
  APP --> LB
  API --> LB
  LB --> GW
  GW --> INT
  INT --> RW
  RW --> DEC
  DEC --> QC
  QC -. miss .-> MERGE
  MERGE --> WEB
  MERGE --> VEC
  MERGE --> RT
  WEB --> BM25
  VEC --> EMB
  RT --> EMB
  BM25 --> XENC
  EMB --> XENC
  XENC --> DEDUP
  DEDUP --> READ
  READ --> SYN
  SYN --> COV
  COV --> SSE
  COV --> FU
  FU --> SSE
  SSE --> UI
  WEB --> CRAWL
  VEC --> DOCS
  SYN --> ANS
  GW -.events.-> EVT
  EVT --> OBS

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class UI,APP,API client;
    class LB,GW,SSE edge;
    class INT,RW,DEC,MERGE,READ,SYN,FU service;
    class BM25,EMB,XENC,DEDUP,COV compute;
    class QC cache;
    class WEB,VEC,CRAWL,DOCS,ANS datastore;
    class RT external;
    class EVT queue;
    class OBS obs;

Query understanding#

The planner is the first LLM call on the critical path, so it runs on a small, fast model (Haiku class, ~50 ms target). It does three things in a single structured-output call: classify intent, rewrite the user's phrasing, and decide whether to decompose.

flowchart TB
  Q[User query]
  CLEAN[Normalize<br/>lowercase, strip filler]
  INT{Intent}
  FACT[Factual lookup]
  NEWS[News / time sensitive]
  COMP[Comparison]
  HOW[How-to]
  NAV[Navigational]
  CODE[Code question]
  RW[Rewrite for search<br/>SEO-friendly]
  DEC{Compound?}
  SUB["Decompose to N<br/>sub-queries"]
  SINGLE[Single query]
  OUT[Plan: sub-queries + intent]

  Q --> CLEAN
  CLEAN --> INT
  INT --> FACT
  INT --> NEWS
  INT --> COMP
  INT --> HOW
  INT --> NAV
  INT --> CODE
  FACT --> RW
  NEWS --> RW
  COMP --> RW
  HOW --> RW
  NAV --> RW
  CODE --> RW
  RW --> DEC
  DEC -- yes --> SUB
  DEC -- no --> SINGLE
  SUB --> OUT
  SINGLE --> OUT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class Q,OUT service;
    class CLEAN,INT,RW,DEC,SUB,SINGLE compute;
    class FACT,NEWS,COMP,HOW,NAV,CODE compute;

Intent matters because it routes the retrieval. News intent skips the cached vector store and goes straight to a real-time fetcher plus a news-specific API. Factual intent prefers the cached web index. Code intent adds Stack Overflow and GitHub to the source list. Navigational intent ("Twitter login") usually just returns the URL and skips synthesis.

Decomposition is where naive systems lose quality. "Compare React, Vue, and Svelte for a beginner in 2026" is three sub-queries (one per framework) plus a synthesis step that compares; running it as a single query buries the smaller frameworks under React noise. The decomposer caps at 4 sub-queries to bound cost.

Retrieval layer#

Three sources run in parallel for every query. The merger collects results from all three, normalizes them into a single candidate list keyed by URL, and hands the merged list to the ranking tier.

flowchart LR
  Q[Sub-query]
  WEB[Web index<br/>own crawl<br/>or Bing / Brave / Tavily]
  VEC[(Vector store<br/>50M cached docs)]
  RT[Real-time fetcher<br/>top SERP URLs]
  M[Merger<br/>dedup by URL]
  OUT[Candidate set<br/>~50 docs]

  Q --> WEB
  Q --> VEC
  Q --> RT
  WEB --> M
  VEC --> M
  RT --> M
  M --> OUT

    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    class Q,OUT service;
    class WEB datastore;
    class VEC datastore;
    class RT external;
    class M compute;

Web index. The hardest piece to build from scratch. Most AI search startups skip the crawl and pay a Bing Search API or Brave Search API wholesale (around $5 per 1k queries at scale), which returns the same SERP that powers the consumer search engine. Building your own crawl plus inverted index is a multi-year, multi-billion dollar investment; Perplexity built its own only after raising 9-figure rounds. For the interview, either path is acceptable; defaulting to a BSP (Bing-style provider) is the realistic answer.

Vector store of cached docs. When the system fetches a page, it cleans the HTML, chunks the text, embeds the chunks, and caches them in a vector store keyed by URL. Future queries hit this cache via approximate nearest neighbor search. Hit rate for popular queries is high (60 to 80%), because most queries cluster around the same evergreen content.

Real-time fetcher. For freshness-sensitive intents, the fetcher takes the top 5 to 10 URLs from the web index, fetches them right now over HTTP, runs a readability extractor (Mozilla readability, trafilatura) to strip nav and ads, and pushes the cleaned text into the candidate set. This is the path that makes "what just happened" queries work.

Race condition matters: the real-time fetch is slow (200 to 800 ms per URL), so the system fetches in parallel up to a wall clock budget. Whatever returns by 1 second goes into ranking; the rest is dropped.

Reranking#

The merged candidate set is too large and too noisy to feed directly to the synthesizer. Ranking happens in two stages: a cheap lexical first pass (BM25) to narrow down, then a cross-encoder reranker on the top 50 to produce the final top 5 to 10.

A cross-encoder reads the query and a candidate passage together and outputs a single relevance score. This is more accurate than dual-encoder embedding similarity but is too expensive to run on millions of docs; that's why BM25 narrows first. See reranking for the full deep dive on this pattern.

flowchart TB
  CAND["Candidate set<br/>~50 to 100 docs"]
  BM25[BM25 + features<br/>top 50]
  EMB[Embed query + doc]
  XENC[Cross-encoder rerank<br/>top 10]
  DEDUP[Domain dedup<br/>max 2 per domain]
  TRUST[Source trust score<br/>downweight low-trust]
  FRESH[Freshness boost<br/>recent for news intent]
  TOP[Top 5 to 8 passages<br/>to reader]

  CAND --> BM25
  BM25 --> EMB
  EMB --> XENC
  XENC --> DEDUP
  DEDUP --> TRUST
  TRUST --> FRESH
  FRESH --> TOP

    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class CAND,TOP service;
    class BM25,EMB,XENC,DEDUP,TRUST,FRESH compute;

Domain dedup is critical. Without it, the same news story reposted on five sites floods the top results and the synthesizer ends up citing five copies of the same sentence. The cap is usually 2 results per domain, with a higher cap for primary sources (official documentation, government sites).

Trust scoring is a learned signal that downweights known low-quality domains (content farms, AI-generated spam sites, scraper aggregators). Perplexity maintains its own domain reputation table; cheaper systems borrow signals from web search APIs that already do this.

Reader model#

The reader is a small, cheap model that takes each top-ranked document and extracts the 1 to 3 passages most relevant to the query. The reader does not write the final answer; it produces structured passages with metadata.

class ReaderOutput:
    url: str
    title: str
    passages: list[Passage]

class Passage:
    text: str            # 1 to 3 sentences, verbatim from the doc
    char_start: int      # offset for citation linkback
    char_end: int
    relevance: float     # reader confidence

The reader runs on a small model (Haiku, GPT-4o-mini class) at low temperature. Output is structured JSON with verbatim passages (not paraphrased), so the synthesizer's later citation linkback can highlight the exact text on the source page.

This stage is what turns "we have 8 web pages" into "we have 12 grounded passages with character offsets". The synthesizer's job becomes much easier: it composes from a small set of pre-extracted facts rather than from raw HTML.

Synthesizer#

The synthesizer is a frontier model (Sonnet, GPT-4 class) that writes the final answer. It receives the original query, the user's conversation history if any, and the list of reader passages with their URL and passage IDs. It writes a 1 to 3 paragraph grounded answer, citing each claim with a numbered marker.

sequenceDiagram
  participant U as User
  participant SYN as Synthesizer
  participant CTX as Context
  participant SSE as Stream
  participant CHK as Coverage checker

  U->>SYN: query + passages
  SYN->>CTX: build prompt<br/>passages with [P1]..[Pk] markers
  Note over SYN: System rule cite every claim<br/>with Pk, refuse if no source
  SYN-->>SSE: stream tokens
  SSE-->>U: first token at ~1.5s
  loop tokens
    SYN-->>SSE: token
    SSE-->>U: token
  end
  SYN->>CHK: final answer + claim spans
  CHK-->>CHK: each sentence has cite?
  alt all covered
    CHK-->>U: render with refs
  else missing cites
    CHK-->>SYN: rewrite uncovered claim
  end

The prompt to the synthesizer is engineered around two rules: every factual sentence ends with one or more [Pk] markers; if no passage supports a claim, refuse to make that claim. The model is good at the first rule on its own; the second needs a post-generation check (see citation coverage below).

The output is streamed token by token through Server-Sent Events back to the UI. The UI rewrites the inline [Pk] markers into clickable numbered citations as they arrive. Hover or click reveals the source URL, title, and the verbatim passage text the claim was grounded on.

Citation post-processing#

The model emits inline [P3] style markers tied to the passages it received. The post-processor maps those passage IDs to citation numbers and source URLs, deduplicates citations that point at the same URL, and replaces the markers with the final numbered references.

import re

CITE_PATTERN = re.compile(r"\[P(\d+)\]")

def post_process_citations(answer_text: str, passages: list[Passage]) -> tuple[str, list[Source]]:
    """Map [P3] markers to [1], [2], ..., deduped by URL."""
    seen_urls: dict[str, int] = {}
    sources: list[Source] = []

    def repl(match: re.Match) -> str:
        passage_idx = int(match.group(1))
        if passage_idx >= len(passages):
            return ""
        url = passages[passage_idx].url
        if url not in seen_urls:
            seen_urls[url] = len(sources) + 1
            sources.append(Source(
                num=seen_urls[url],
                url=url,
                title=passages[passage_idx].title,
                snippet=passages[passage_idx].text,
            ))
        return f"[{seen_urls[url]}]"

    rewritten = CITE_PATTERN.sub(repl, answer_text)
    return rewritten, sources


def coverage_check(answer_text: str, sources: list[Source]) -> list[str]:
    """Return a list of sentences with no citation marker."""
    uncovered = []
    sentences = re.split(r"(?<=[.!?])\s+", answer_text)
    for sent in sentences:
        if not re.search(r"\[\d+\]", sent) and _is_factual(sent):
            uncovered.append(sent)
    return uncovered

The coverage check is the guardrail. Any factual sentence that ended up with no [N] marker is flagged. The system has two options: re-prompt the synthesizer to either add a citation or remove the claim, or simply strip the uncovered sentence before returning the answer. Production systems do both and pick whichever yields the cleaner final answer.

Follow-up generation#

After the answer streams, a small model runs a single call with the original query and the answer (not the full passages, to keep cost low) and emits 5 follow-up question candidates. A diversity filter scores them by semantic distance from the original query and from each other, dropping near-duplicates, and the top 3 are returned to the UI.

flowchart LR
  ANS[Final answer<br/>+ original query]
  GEN[Small model<br/>emit 5 followups]
  DIV[Diversity filter<br/>embed + dedup]
  SCORE[Score by<br/>novelty + coherence]
  TOP3[Top 3 followups]

  ANS --> GEN
  GEN --> DIV
  DIV --> SCORE
  SCORE --> TOP3

    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    class ANS,TOP3 service;
    class GEN,DIV,SCORE compute;

Good follow-ups extend the user's exploration rather than restate the original answer. The prompt is engineered to favor "what about X next" and "how does this compare to Y" patterns over "tell me more about Z" patterns that produce redundant clicks.

Freshness and real-time crawl#

Most queries do not need real-time data. "What is consistent hashing" can be served from a 6-month-old crawl with no loss in quality. The system decides query by query whether to pay the latency cost of fetching now.

Intent Cache TTL Real-time fetch?
Definition / encyclopedic 30 days No
Documentation / tutorial 7 days No
Comparison / how-to 7 days No
Product info / pricing 1 day Sometimes
News / current events 5 minutes Yes
Sports / stock / weather 1 minute Yes
Government / legal data 1 hour Sometimes
Code questions 1 day No (Stack Overflow API)

The intent classifier sets a cache TTL. If the cached vector store has a fresh-enough entry for the query, retrieval skips the real-time fetch; otherwise the fetcher races against the wall clock budget. This is the single biggest cost lever: cached queries cost roughly 10x less than queries that trigger a real-time fetch fanout.

Hallucination guardrails#

A grounded answer engine has three layered guardrails against hallucination:

  1. Reader extracts verbatim, not paraphrased. The passages handed to the synthesizer are exact text from the source page, with character offsets. The synthesizer can quote or restate, but the supporting text is verifiable.
  2. Citation coverage check after generation. Every factual sentence must have at least one [N] marker. Uncovered sentences are flagged and either rewritten or removed.
  3. Source-claim verifier (optional, second-pass). A cheap model takes each cited claim and the cited passage, and answers "does this passage support this claim, yes or no". Failures get flagged. This runs only on a sample of high-stakes queries in production due to cost.

Refusal is a feature, not a failure. If the candidate set is empty or all candidates have low rerank scores, the synthesizer emits "I don't have a confident answer for this" instead of writing speculative text. Perplexity's product specifically advertises this behavior; chat-only LLMs without retrieval often hallucinate here.

Cost and latency#

The pipeline has four LLM calls (planner, reader, synthesizer, follow-up) plus retrieval fan-out. Sub-second first token requires aggressive parallelism and tight model selection.

gantt
    title Critical path for one query (target p50 first token 1.5s)
    dateFormat X
    axisFormat %Lms
    section Plan
    Intent classifier      :a1, 0, 100
    Rewriter + decomposer  :a2, after a1, 150
    section Retrieve
    Web index call         :b1, after a2, 250
    Vector store call      :b2, after a2, 200
    Real-time fetch (parallel) :b3, after a2, 800
    section Rank
    BM25 + dedup           :c1, after b1, 50
    Cross-encoder rerank   :c2, after c1, 150
    section Read + synth
    Reader extract         :d1, after c2, 200
    Synthesizer first token :d2, after d1, 300

The longest path is the real-time fetcher (800 ms), but it runs in parallel with the rest, so it is rarely the bottleneck unless the wall-clock budget runs out. The serial dependencies that actually bound latency are: plan -> rank -> read -> synth, which is around 1.0 to 1.5 seconds to first token in a warm system.

Cost per query at scale (after caching kicks in):

Stage Cost component Per-query cost
Planner Small model, ~500 in / 100 out $0.0002
Retrieval BSP API or own index $0.0050
Rerank Cross-encoder GPU time $0.0005
Reader Small model, ~3k in / 400 out $0.0015
Synthesizer Frontier model, ~5k in / 300 out $0.0050
Follow-up gen Small model, ~500 in / 100 out $0.0002
Total ~$0.012

This is before caching. With normalized-query caching (about 30% hit rate for popular queries), the blended cost lands near $0.008 per query, which is the order of magnitude Perplexity targets for its free tier and slightly less for its paid Pro tier.

Comparison: Perplexity, ChatGPT browse, Google AI Overviews#

Dimension Perplexity ChatGPT browse Google AI Overviews Bing Chat
Primary surface Answer engine Chat with optional browse SERP feature above blue links Sidebar in Bing search
Citation prominence Inline numbered, mandatory Lighter, sometimes absent Linked summary card Inline footnotes
Source Own index + BSP partners Bing API Google's full search index Bing's full index
Default model Sonnet / GPT-4 class GPT-4 class Gemini GPT-4 class
Follow-ups First-class feature None Limited None
Refusal Strong (advertised) Weak Moderate Moderate
Latency target First token 1.5 s 3 to 5 s Inline with SERP 2 to 4 s
Differentiator UX wrapper around the pattern Chat-first Tight coupling to Google index Web access for ChatGPT users

Perplexity's moat is execution, not architecture: every AI search system uses the same pattern. The differentiator is the planner's intent classification quality, the reranker's domain-trust signal, the cache hit rate on popular queries, and the UI polish around citations and follow-ups.

Real-world numbers#

  • Perplexity processed ~250M queries per month by mid-2024 and crossed ~1B per month by late 2025; growth roughly tracks the broader AI search shift.
  • Query latency budget: first token 1.5 s p50, ~5 s full answer p95.
  • Citation precision benchmarks: Perplexity's product cites 95%+ precision in internal benchmarks; independent academic benchmarks (HaluEval, FActScore variants) put grounded systems at 80 to 90% claim-level precision depending on the eval set.
  • Cost per query: industry estimates put consumer AI search systems at $0.005 to $0.015 per query at scale, dominated by search API fees and synthesizer tokens.
  • Free-tier limits are usually around 5 to 10 "deep" queries per day per user, with cheaper-model variants for unlimited daily queries.

Common interview tradeoffs#

The interviewer will press on these. Pre-canned answers:

Tradeoff What to say
Own crawl vs BSP API Own crawl is a multi-year, multi-billion dollar investment; BSP gets you to product faster at a higher per-query cost. Hybrid is realistic: BSP for tail, own crawl for high-traffic topics.
Single-model vs tiered Tiered: small model for plan / reader / followup, frontier for synthesis. Reduces per-query cost by 3 to 5x with negligible quality drop in user studies.
Sync rerank vs async Sync. The reranker is on the critical path; pushing it async hurts answer quality more than it helps latency.
Cite count: 3 vs 10 3 to 5 is the sweet spot. More citations confuse the user and dilute click-through; fewer hides supporting evidence.
Real-time fetch always vs on-demand On-demand. Always-fetch triples per-query cost and adds 500 to 800 ms of latency on queries that don't need fresh data.
Streaming the answer vs waiting Stream. Waiting feels broken at 4 seconds; streaming hides the latency by getting tokens in front of the user at 1.5 s.
Refusal vs always-answer Refusal. Hallucinated cites destroy trust faster than empty answers; the product brand is "grounded answers, not chatbot guesses".

Common interview follow-ups#

  • "How would you crawl the web for this? What's the size?" Answer with the web crawler deep dive: distributed crawler, URL frontier, politeness, dedup, freshness scheduling, sizing to ~100B URLs.
  • "How does the vector store stay fresh?" Crawl emits cleaned documents to a chunking + embedding pipeline; vectors land in the store with a TTL based on intent class.
  • "What if the BSP rate-limits you?" Multi-provider router with failover (Bing -> Brave -> Tavily); cache the BSP response by normalized query.
  • "How do you handle non-English queries?" Plan in the user's language; rewrite for search in the most common index language for that region; synthesize in the original language.
  • "How do you evaluate citation precision at scale?" Sampled human eval plus an LLM-as-judge for cheap continuous monitoring; track precision per (intent class, domain) bucket so regressions are visible.

FAQ#

How does Perplexity actually work under the hood?#

A planner rewrites the user's query into one or more sub-queries, fans out to a web index plus a real-time fetcher, reranks the candidate pages, extracts supporting passages, and a synthesizer model writes an answer with inline numbered citations linking back to those passages.

How is AI search different from naive RAG?#

Naive RAG retrieves from a static private corpus; AI search retrieves from the live public web, must handle freshness, must cite external URLs, and must refuse claims that no retrieved source supports. The synthesizer also has to generate follow-up questions for exploration.

How does Perplexity prevent hallucinations?#

Every claim in the answer must trace to a cited source. A citation coverage check runs after generation, flagging any sentence with no supporting passage. Unsupported claims are rewritten or removed, and on weak evidence the system abstains.

How does AI search stay fast under a multi-step pipeline?#

Aggressive parallel fetch across retrieval sources, a normalized-query cache, streaming synthesis that starts emitting tokens as the first reranked passages arrive, and a tiered model strategy where cheap models handle planning and reranking.

When should I bypass the cache and crawl in real time?#

When the query intent is time sensitive: news, stock prices, sports scores, live events. Intent classification at planning time routes those to a real-time fetcher; static knowledge queries (history, definitions) hit the cached web index.

What is the difference between Perplexity, ChatGPT browse, and Google AI Overviews?#

Perplexity puts citations first and writes a short grounded answer; ChatGPT browse adds optional web fetch to a chat model with fewer citations; Google AI Overviews summarizes the top SERP results above the blue links and stays tightly coupled to Google's index.

How are follow-up questions generated?#

The same synthesizer model produces 3 to 5 short related questions per answer using the gathered evidence as context, scoring them for diversity and conversational coherence before showing the top candidates.

  • RAG Patterns: naive, advanced, modular RAG patterns the AI search pipeline specializes
  • Hybrid Search: BM25 + Vector: the lexical plus dense retrieval combination that powers the ranking tier
  • Reranking: cross-encoder reranker design used between retrieval and reader
  • Chunking Strategies: how cleaned web pages get split before vector embedding
  • Web Crawler: the crawl + index infrastructure behind a self-hosted web source

Quick reference#

Functional#

  • NL question in, grounded answer with inline citations out.
  • Intent classification: factual, news, comparison, how-to, navigational, code.
  • Query rewrite + compound decomposition (cap 4 sub-queries).
  • Three retrieval sources in parallel: web index, vector cache, real-time fetch.
  • Cross-encoder rerank with domain dedup and trust scoring.
  • Reader extracts verbatim passages with char offsets.
  • Synthesizer writes grounded answer with [N] inline cites.
  • Citation coverage check; refuse on weak evidence.
  • 3 to 5 follow-up questions per answer.
  • Multi-turn conversation reusing prior evidence.

Non-functional#

  • p50 first token 1.5 s, p95 full answer 6 s.
  • 100k QPM peak, 5k QPS sustained.
  • 95% citation precision.
  • 99.9% availability.
  • Freshness: news under 5 min old.
  • Cost target $0.01 per query at scale.

Retrieval signals#

Signal Source Weight role
BM25 score Web index First-pass narrowing
Cross-encoder score Reranker Final ranking
Domain trust score Reputation table Downweight low-quality
Freshness boost Crawl timestamp News / time-sensitive intent
Click signal (long-term) User logs Domain reputation feedback
Source authority Curated list Boost primary sources (gov, docs)
Per-domain cap Diversity rule Max 2 results per domain

Citation formats#

Style Example Used by
Inline numbered The election margin was 2.4% [1][2]. Perplexity, default
Inline footnote The election margin was 2.4%.[^1] Markdown-export mode
Floating card Hover over claim shows source card UI overlay
End-of-section list All sources gathered at section end Long-form reports
Linked phrase The election margin was [2.4%](URL) ChatGPT browse

Latency budget per stage#

Stage Budget p50 Notes
Intent classify 80 ms Small model, structured output
Rewrite + decompose 150 ms Same call as classify possible
Web index call 250 ms BSP API or own index
Vector store ANN 200 ms Parallel with web index
Real-time fetch 800 ms Parallel; capped at wall clock
BM25 narrowing 50 ms In-process
Cross-encoder 150 ms GPU inference
Reader extract 200 ms Small model
Synthesizer TTFT 300 ms Frontier model first token
Follow-up gen 200 ms Parallel with last synth tokens
First token p50 ~1.5 s Plan + retrieve + rank + read + first synth token
Full answer p95 ~6 s Adds full synth body + coverage check

Cost budget per query#

Stage Model class Approx cost
Planner Small (Haiku, GPT-4o-mini) $0.0002
Retrieval BSP API $0.0050
Rerank Cross-encoder, batched $0.0005
Reader Small model $0.0015
Synthesizer Frontier (Sonnet, GPT-4) $0.0050
Follow-up gen Small model $0.0002
Per-query total ~$0.012
After 30% cache hit ~$0.008

Common pitfalls#

  • No domain dedup; same story cited 5 times from copycat sites.
  • Skipping the citation coverage check; hallucinated cites slip through.
  • Single retrieval source; news queries return stale data, factual queries return shallow data.
  • Synthesizer over a single search API result; cite precision drops below 80%.
  • Always-on real-time fetch; cost triples for no quality gain on evergreen queries.
  • No refusal path; product brand collapses when the model invents sources.
  • Streaming the answer but blocking citations until end; UX feels broken when refs pop in 4 seconds late.
  • Cross-encoder reranker on every candidate; bill explodes. BM25 must narrow first.
  • Decomposing every query; simple lookups become 4x slower for no benefit.
  • Caching by raw query string; misses 80% of near-duplicates. Normalize first.

Refs#

  • Perplexity engineering blog posts on architecture and citation precision.
  • Anthropic, OpenAI, and Google research blogs on grounded generation and hallucination benchmarks.
  • Bing Search API, Brave Search API, and Tavily public documentation.
  • Mozilla readability, trafilatura, jusText for HTML extraction.
  • HaluEval, FActScore, FreshQA evaluation suites.
  • Academic surveys on retrieval-augmented generation and dense retrieval (DPR, ColBERT, SPLADE).