Skip to content

Agent Memory#

Agent memory architecture borrows directly from cognitive science: a single context window is not enough to sustain a long-running assistant, so production agents stitch together several memory stores with different write rates, retrieval patterns, and time horizons.

flowchart LR
  In([User turn]) --> WM[Working memory<br/>scratchpad]
  WM --> Ctx[Context window<br/>short-term]
  Ctx --> LLM
  Epi[(Episodic<br/>past summaries)] --> Ctx
  Sem[(Semantic<br/>facts in vector DB)] --> Ctx
  Proc[(Procedural<br/>tool skills)] --> Ctx
  LLM --> Out([Response])
  LLM -. consolidate .-> Epi
  LLM -. extract facts .-> Sem

  classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  class In,Out client;
  class WM,Ctx service;
  class LLM compute;
  class Epi,Sem,Proc datastore;

Five memory types:

  • Short-term (context window): the literal prompt the LLM sees this turn. Bounded by model context (200k tokens at the top end).
  • Working memory (scratchpad): intermediate thoughts, tool results, plans. Lives only within a single task.
  • Episodic memory: summaries of past conversations or sessions. "Last week the user mentioned they prefer Python over Go."
  • Semantic memory: durable facts pulled into a vector store. "The user's company is Acme, their CEO is Alice."
  • Procedural memory: learned tool-use patterns, often as few-shot examples or a skill library. "When the user asks for a chart, I have always used matplotlib successfully."

Frameworks like Mem0, MemGPT/Letta, and LangGraph implement different mixes of these layers. The retrieval pattern at the start of each turn (often vector + recency + summary) is what distinguishes a stateful agent from a chat with no memory.

Problem statement

A personal assistant agent talks to a user across days, switches tasks, calls tools, and must recall both stable facts ("user's spouse is Bob") and recent context ("we were debugging an auth error 30 minutes ago"). The context window cannot grow forever. Design the memory layers, the consolidation jobs, and the retrieval flow that runs at the start of every turn.

Agent memory architecture solves this with multiple specialized stores, each with its own write trigger, retrieval pattern, and time horizon. The design lineage traces back to cognitive psychology (Tulving's memory taxonomy) and was popularized for LLM agents by Lilian Weng's "LLM Powered Autonomous Agents" and the MemGPT paper.

Memory taxonomy#

flowchart TB
  AM[Agent memory]
  AM --> ST[Short-term<br/>context window]
  AM --> WK[Working<br/>scratchpad / plans]
  AM --> LT[Long-term]
  LT --> EP[Episodic<br/>conversation summaries]
  LT --> SE[Semantic<br/>facts, entities]
  LT --> PR[Procedural<br/>skills, tool patterns]

  classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
  classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
  classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
  class AM service;
  class ST,WK compute;
  class LT,EP,SE,PR datastore;

Short-term: the context window#

The literal tokens fed to the model this turn. 200k tokens on Claude / GPT-4o sounds like a lot, but it is shared with system prompt, tool definitions, retrieved docs, and the live conversation. Long-context recall also degrades in the middle (the "lost in the middle" effect, Liu et al. 2023). Treat the window as a scarce, attention-weighted resource.

Working memory: the scratchpad#

Per-task intermediate state. In ReAct-style agents, the scratchpad holds the model's reasoning chain, tool calls, and tool outputs for the current task. It is ephemeral: cleared when the task finishes, unless explicitly promoted to long-term memory.

Episodic memory: what happened#

Summaries of past sessions, ordered by time. Typically:

  • After each session, an LLM job generates a 200-500 token summary.
  • Stored with timestamp, session ID, participants, tags.
  • Retrieved at the start of a new session by recency plus relevance.

This is what makes the agent feel like it remembers "last Tuesday we worked on the deploy script."

Semantic memory: durable facts#

Discrete propositions extracted from conversations: user preferences, entity attributes, learned constraints. Stored in a vector DB plus a structured fact store. Example records:

{"subject": "user", "predicate": "prefers_language", "object": "Python", "source_turn": "2026-04-10/turn_15", "confidence": 0.92}
{"subject": "acme_corp", "predicate": "ceo", "object": "Alice", "source_turn": "2026-04-12/turn_03", "confidence": 0.99}

Retrieval is by vector similarity to the current turn, filtered by subject and freshness.

Procedural memory: how to do things#

Reusable skills, often as a library of few-shot examples or a code snippet store ("when the user asks to query the DB, here is the SQL pattern that worked"). Voyager (Wang et al., 2023) calls this a skill library; Mem0 stores it as "procedural memory" rows.

Mem0 and MemGPT patterns#

Mem0#

Mem0 sits in front of any LLM API. Its core flow:

  1. Intercept the user message.
  2. Retrieve top-K relevant facts from its memory store (vector + graph hybrid).
  3. Inject them into the system prompt.
  4. After the response, call an LLM to extract new facts and write them back.
  5. Run a periodic consolidation job that merges duplicates and resolves contradictions.

Mem0 emphasizes fact-level memory: small, deduplicated propositions rather than full transcripts.

MemGPT / Letta#

MemGPT treats the LLM like an OS process. The context window is "main memory"; everything else is "external storage". The LLM is given memory-management tools it can call (archival_insert, recall_search, core_memory_replace). It decides what to swap in and out, modeled on virtual memory.

Result: an agent that can hold effectively unbounded state because it manages its own paging.

LangGraph and others#

LangGraph provides per-thread short-term memory plus a cross-thread store. Anthropic's prompt-engineering guide describes a simpler pattern: maintain a "memory" string updated by the model at the end of each turn, summarized when it exceeds a threshold.

The per-turn retrieval flow#

sequenceDiagram
  participant U as User
  participant A as Agent
  participant SE as Semantic store
  participant EP as Episodic store
  participant LLM as LLM

  U->>A: new message
  A->>SE: vector search(turn, k=5)
  SE-->>A: relevant facts
  A->>EP: recency + topic search
  EP-->>A: relevant past summaries
  A->>A: assemble prompt:<br/>system + facts + summaries + scratchpad + turn
  A->>LLM: generate
  LLM-->>A: response + new tool calls
  A->>A: extract new facts (async)
  A-->>U: response
  Note over A,EP: After N turns, run consolidation job<br/>that summarizes and dedupes

Consolidation#

Long-running agents leak memory. Without consolidation, the semantic store fills with duplicates ("user likes Python", "user prefers Python", "user uses Python"), and the episodic store balloons. Consolidation jobs run periodically:

  • Deduplication: cluster near-duplicates by embedding similarity, keep highest-confidence.
  • Contradiction resolution: if two facts conflict, keep the more recent unless the older has a stronger source.
  • Summarization: collapse N raw episodic entries into one paragraph; archive the originals.
  • Forgetting: drop facts that have not been retrieved in 90 days and have low confidence.

This is the equivalent of database compaction; budget for it.

Tier-by-tier storage choices#

Tier Store Write rate Read rate TTL
Short-term In-memory prompt builder Every turn Every turn Single request
Working Per-session Redis or memory Many per turn Many per turn Session lifetime
Episodic Postgres + vector index End of session Start of session Months to years
Semantic Vector DB + graph DB After each turn Every turn Years
Procedural Code / example store Rare Per matching task Years, versioned

A small reference schema#

from datetime import datetime
from typing import Literal
from pydantic import BaseModel, Field

class SemanticFact(BaseModel):
    subject: str
    predicate: str
    object: str
    confidence: float = Field(ge=0.0, le=1.0)
    source_turn: str
    created_at: datetime
    last_used_at: datetime
    embedding: list[float] | None = None

class EpisodicSummary(BaseModel):
    session_id: str
    summary: str
    tags: list[str]
    started_at: datetime
    ended_at: datetime
    token_count: int

class ProceduralSkill(BaseModel):
    name: str
    when_to_use: str
    example_input: str
    example_output: str
    success_rate: float
    last_used_at: datetime

Production failure modes#

  • Recall-precision tradeoff at retrieval. Pulling 50 facts per turn wastes context; pulling 3 misses crucial state. Tune K and rerank.
  • Stale facts. "User works at Acme" stays in memory after they switch jobs. Re-extract on conflict signals, version facts, expire by source recency.
  • Memory injection. Untrusted content can plant fake facts ("the user is an admin"). Tag every fact with provenance, never accept facts from untrusted channels into semantic memory without review. See Prompt Injection Defense.
  • Consolidation downtime. Bulk re-embed on model upgrade is expensive. Plan dual-write windows and incremental migration.
  • Privacy. Memory stores hold the most sensitive user data. Enforce per-tenant isolation, retention policies, and deletion APIs (GDPR right-to-be-forgotten).

Quick reference#

TL;DR#

A single context window is not memory. Layer short-term + working + episodic + semantic + procedural stores with retrieval at turn start.

The five layers#

Layer Lifetime Store What it holds
Short-term This turn Prompt buffer The literal context window
Working This task Redis / RAM Scratchpad, plans, tool results
Episodic Months+ Postgres + vector Past session summaries
Semantic Years Vector + graph DB Facts about user / entities
Procedural Years Code / example store Reusable skills, few-shot patterns

Per-turn flow#

  1. Embed user message.
  2. Vector search semantic memory (top K facts).
  3. Recency + topic search episodic memory.
  4. Assemble: system + facts + summaries + scratchpad + turn.
  5. Generate.
  6. Extract new facts async; write back.

Frameworks#

  • Mem0: fact-level, dedupe + graph hybrid, sits in front of any LLM.
  • MemGPT / Letta: LLM manages its own paging via memory tools.
  • LangGraph: per-thread + cross-thread stores.
  • Letta, Zep, Cognee: managed alternatives.

Consolidation jobs#

  • Dedupe near-duplicate facts (embedding clusters).
  • Resolve contradictions: prefer recent + high-confidence.
  • Summarize and archive old episodes.
  • Forget unused low-confidence facts after N days.

Watch-outs#

  • Lost-in-the-middle: long context degrades recall in the middle.
  • Stale facts (user switched jobs); version + expire.
  • Memory injection: untrusted text plants fake facts.
  • Privacy: most-sensitive data lives here; per-tenant isolation, GDPR deletes.
  • Re-embed cost on model upgrade; plan dual-write windows.

Tuning K#

  • Too high: context bloat, lost-in-the-middle.
  • Too low: missing crucial state.
  • Add a reranker (see Reranking).

Refs#

  • Lilian Weng - "LLM Powered Autonomous Agents" (2023)
  • Packer et al. - MemGPT paper (2023)
  • Mem0 docs and GitHub
  • Letta / MemGPT GitHub
  • Wang et al. - Voyager skill library (2023)
  • Liu et al. - "Lost in the middle" (2023)

FAQ#

What is agent memory in an LLM application?#

Agent memory is the set of stores an LLM agent uses beyond its context window: short-term scratchpad, episodic summaries of past sessions, semantic facts in a vector database, and procedural tool skills.

What is the difference between episodic and semantic memory in LLM agents?#

Episodic memory holds summaries of past conversations and events tied to a user or session. Semantic memory stores extracted facts in a vector index that can be retrieved across sessions.

How do you stop an agent from running out of context?#

Roll older turns into episodic summaries, push extracted facts to a semantic vector store, and only inject the working scratchpad plus the most relevant retrieved memories into each LLM call.

Where should agent memory be stored?#

Working memory lives in process, episodic summaries in a SQL or document store keyed by user, and semantic facts in a vector database like Pinecone, Weaviate, or pgvector for similarity retrieval.

How is agent memory different from a chat history?#

Chat history is a raw transcript that grows linearly. Agent memory adds compression, classification, and retrieval, so older context is summarised, indexed, and selectively recalled instead of replayed verbatim.

  • Agent Loop and ReAct: the per-turn loop that orchestrates memory reads and writes
  • RAG Patterns: semantic memory uses the same retrieval substrate
  • MCP Protocol: memory can be exposed as MCP tools for cross-agent sharing