Ad Click Aggregation / Ad Server#

Q: How is ad click aggregation done at scale?

Click events are ingested via Kafka, deduped and validated, then aggregated by stream processors per advertiser, campaign, and time bucket into low-latency stores for dashboards and billing.

Q: How do ad systems prevent click fraud?

Pipelines apply velocity checks, bot signatures, IP reputation, and device fingerprinting before counting clicks. Suspect events are flagged for offline review and billing adjustment.

Q: Why use lambda architecture for ad aggregation?

A real-time stream gives near-instant counts for monitoring, while a batch pipeline produces authoritative end-of-day numbers that reconcile and correct stream errors.

Q: What is ad attribution modeling?

Attribution assigns conversion credit to touchpoints in the user's journey using rules like last-click, first-click, or data-driven models that estimate marginal contribution.

Q: How do ad systems guarantee exactly-once counting?

They use idempotent writes keyed by event ID, transactional sinks, and offset checkpoints in the stream processor to ensure no double-counting after retries.

Q: What metrics matter for an ad aggregation pipeline?

Key metrics include ingest lag, dedupe rate, aggregation latency, dashboard freshness, and reconciliation gap between stream and batch pipelines.

Problem statement (interviewer prompt)

Design a real-time ad-click aggregation system. Ingest billions of click events/day, dedupe + validate (fraud), aggregate per advertiser + campaign + time bucket for billing and dashboards, and support both real-time (seconds) and end-of-day batch reconciliation.

flowchart LR
  U([User])
  ADS[Ad Server]
  AUC[Auction / Bidder]
  CLK[Click / Impression event]
  K[[Kafka]]
  AGG[[Stream Aggregator]]
  RPT[(Reports)]
  U --> ADS --> AUC
  U --> CLK --> K --> AGG --> RPT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class U client;
    class ADS,AUC,CLK service;
    class RPT datastore;
    class K,AGG queue;

flowchart TB
  subgraph Edge[Ad request flow]
    APP[Publisher app / page]
    ADX[Ad Exchange]
    BID[Bidders / DSPs]
    CACHE[Ad creative cache]
    CDN
  end

  subgraph Serve
    SEL[Ad selection]
    PACE[Pacing engine]
    CAP[Frequency cap]
    BUDGET[Budget control]
  end

  subgraph Events[Event pipeline]
    IMP[Impression event]
    CLK[Click event]
    CONV[Conversion event]
    KAFKA[[Kafka]]
    DEDUP[Dedup window]
  end

  subgraph Stream[Stream agg - real-time]
    FLINK([Flink / Spark Streaming])
    HLL[HyperLogLog uniques]
    CMS[Count-Min hot keys]
    WIN[Sliding windows]
  end

  subgraph Batch[Batch - exact]
    SPARK([Spark / Beam jobs])
    LAKE[(Data lake)]
    DAILY([Daily reconciled aggregates])
  end

  subgraph Store
    HOT[(Hot KV: per-ad/per-campaign counters)]
    OLAP[(OLAP: ClickHouse / BigQuery / Druid)]
  end

  subgraph Consumer
    REP[Reporting dashboards]
    BIL[Billing]
    ALERT[Alerting]
  end

  APP --> ADX --> BID --> SEL --> CDN
  APP --> IMP
  APP --> CLK --> KAFKA
  IMP --> KAFKA
  CONV --> KAFKA
  KAFKA --> Stream --> HOT --> REP
  KAFKA --> Batch --> OLAP --> BIL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP,ADX,BID,CACHE,SEL,PACE,CAP,BUDGET,IMP,CLK,CONV,DEDUP,HLL,CMS,WIN,BIL service;
    class LAKE,HOT,OLAP datastore;
    class KAFKA queue;
    class FLINK,SPARK,DAILY compute;
    class REP,ALERT obs;

Lambda / Kappa architecture#

Stream layer: fast, approximate counts (HLL, CMS).
Batch layer: exact, idempotent, reconciles within minutes/hours.
Serving merges both: real-time first, batch overwrites for accuracy.

Anti-fraud (clicks)#

Bot / non-human traffic filters.
Per-ip / per-cookie velocity, device fingerprint, conversion rate sanity.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
`HLD`	CDN	edge caching for static assets	cdn
`HLD`	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
`HLD`	CAP / PACELC	C vs A under partition; L vs C otherwise	cap-pacelc
`HLD`	Probabilistic data structures	Bloom, HLL, Count-Min, MinHash, t-digest	probabilistic-data-structures
`HLD`	Batch & stream processing	Lambda vs Kappa, watermarks, windows	batch-stream-processing

Quick reference#

Functional#

Real-time counts: impressions, clicks, conversions per ad/campaign.
Budget pacing & shut-off.
Fraud / IVT filtering.
Reporting dashboards + billing.

Non-functional#

Real-time aggregates < 5 s lag.
Daily reconciled exact totals.
99.9% event durability.

Capacity#

Trillions of events/yr at top platforms.
HLL keeps cardinality cheap; CMS keeps hot keys cheap.

Trade-offs#

Streaming approximate vs batch exact: combine in lambda/kappa.
At-least-once + dedup at sink is simpler than exactly-once everywhere.
Storage: column store for OLAP; KV for per-ad fast lookups.

Refs#

Google Ad Manager engineering posts.
Facebook "Realtime data processing" blogs (Scribe, Puma).
Druid / Pinot for ad analytics.

FAQ#

How is ad click aggregation done at scale?#

Click events are ingested via Kafka, deduped and validated, then aggregated by stream processors per advertiser, campaign, and time bucket into low-latency stores for dashboards and billing.

How do ad systems prevent click fraud?#

Pipelines apply velocity checks, bot signatures, IP reputation, and device fingerprinting before counting clicks. Suspect events are flagged for offline review and billing adjustment.

Why use lambda architecture for ad aggregation?#

A real-time stream gives near-instant counts for monitoring, while a batch pipeline produces authoritative end-of-day numbers that reconcile and correct stream errors.

What is ad attribution modeling?#

Attribution assigns conversion credit to touchpoints in the user's journey using rules like last-click, first-click, or data-driven models that estimate marginal contribution.

How do ad systems guarantee exactly-once counting?#

They use idempotent writes keyed by event ID, transactional sinks, and offset checkpoints in the stream processor to ensure no double-counting after retries.

What metrics matter for an ad aggregation pipeline?#

Key metrics include ingest lag, dedupe rate, aggregation latency, dashboard freshness, and reconciliation gap between stream and batch pipelines.

Video walkthrough

Design an Ad Click Aggregator : via Hello Interview