Spam / Abuse Detection

Problem statement (interviewer prompt)

Design a spam / abuse detection system for a social product (UGC + DMs + comments): score every post / message in <200ms via rules + ML + reputation + behavioural signals, learn from user reports + moderator decisions, and run on adversarial input.

flowchart LR
  E[Event]
  RULE[Rule engine]
  ML([ML classifier])
  ACT[Action: allow / step-up / block]
  E --> RULE --> ACT
  E --> ML --> ACT

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class E,RULE,ACT service;
    class ML compute;

flowchart TB
  subgraph Source
    SIGNUP[Signups]
    MSG[Messages / posts]
    REVIEW[Reviews]
    PAY[Payments]
  end

  subgraph Features
    TEXT([Text features: token n-gram, embeddings])
    BEHAV[Behavior velocity, time-of-day]
    DEVICE([Device fingerprint])
    NET[IP / ASN / VPN detection]
    GRAPH[Account-link graph]
  end

  subgraph Models
    GBDT[GBDT]
    NLP[NLP / transformer]
    ANOM[Anomaly detector]
    SIMHASH[SimHash dedup]
  end

  subgraph Rules
    BLK[Blocklists]
    HARDLIM[Hard limits]
    REGEX[Pattern rules]
  end

  subgraph Decision
    SCORE([Score aggregator])
    POL[Policy]
    EXPL[Explanations]
  end

  subgraph Actions
    ALLOW
    STEP[Step-up: captcha, MFA]
    SHADOW[Shadow ban]
    BLOCK
    QUEUE[Manual review]
  end

  subgraph Loop
    LBL[Labels: reports, chargebacks]
    RETRAIN[Retraining]
  end

  Source --> Features --> Models --> Decision --> Actions
  Rules --> Decision
  Actions --> Loop --> Models

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class DEVICE client;
    class SIGNUP,MSG,REVIEW,PAY,BEHAV,NET,GBDT,NLP,ANOM,SIMHASH,BLK,HARDLIM,REGEX,POL,EXPL,STEP,SHADOW,QUEUE,LBL,RETRAIN service;
    class TEXT,SCORE compute;

Glossary & fundamentals

Concept	What it is	Fundamentals
Probabilistic structures	SimHash dedup	probabilistic-data-structures
Pub/Sub	event bus	pub-sub-pattern
CDC	label updates	change-data-capture
Caching strategies	feature cache	caching-strategies
Resilience patterns	safe defaults under model outage	resilience-patterns
Observability	drift + FPR/FNR	observability

Quick reference

Functional

• Score signups, posts, reviews, messages, payments.
• Rule + ML hybrid.
• Decision: allow / step-up / shadow-ban / block / queue review.
• Feedback loop from reports / chargebacks.

Non-functional

• p99 decision < 100 ms on hot path.
• False positive rate critical (UX impact).
• Adversarial environment - adaptive model freshness.

Trade-offs

• Rules explainable + fast but rigid; ML accurate but opaque.
• Shadow ban vs hard ban: shadow is less hostile but ethically debated.
• Latency budget forces feature pre-computation.

Refs

• "Spam fighting at scale" papers (Akismet, Gmail).
• Yelp / Reddit / X engineering posts.
• ByteByteGo "Design spam detection".

FAQ

How do you design a spam detection system?

Score every post through rules and ML in under 200ms; combine content signals, account reputation, and behavioural patterns. Borderline scores trigger step-up challenges; high scores block silently.

Rules vs ML in spam detection?

Rules catch known patterns precisely and are explainable. ML learns novel abuse but is opaque and needs labelled data. Use rules as a fast first line and ML for everything rules miss.

How is account reputation computed?

A background job aggregates signup quality, account age, prior actions, and report counts into a score. The scorer reads the cached value and updates incrementally as events arrive.

What is a feedback loop in spam systems?

User reports and moderator actions become labels that train the next model version. A shadow model runs in parallel and graduates when its precision and recall beat production.

How do you stop adversaries adapting to filters?

Keep model signals secret, rotate features, randomize challenge timing, and pair ML with reputation so a brand-new account cannot reuse a previously successful exploit.