How AI Agent Memory Works

Every LLM call is stateless by default — the model has no memory of previous conversations beyond what is explicitly included in the current context window. For simple Q&A this is fine, but for agents handling multi-session workflows, this creates a fundamental problem: the agent treats every conversation as if it is the first, forcing users to re-explain their context, preferences, and history repeatedly. At enterprise scale, this is not just annoying — it is a product-killing limitation. A customer service agent that forgets the customer's account type after a context window fills is functionally broken regardless of how capable the underlying model is.

The context window limit is the hard ceiling — 200K tokens for Claude, 128K for GPT-4o — but in practice, context quality degrades before the limit is hit due to the lost-in-the-middle problem. At 100K tokens, information from turn 1 is statistically less attended to than information from turn 100, even though it is technically present. The cost dimension compounds this: a 200K token context at $15/MTok costs $3 per message just for input tokens. Running 10K daily active users at this context size costs $30K/day in input tokens alone. A memory system that compresses context to 5K tokens of the most relevant information reduces this to $750/day — a 40x cost reduction.

Production agent memory systems are architecturally analogous to human memory — a hierarchy of stores with different capacities, speeds, and persistence. In-context memory is working memory: fast and instantly accessible but small and ephemeral. External memory is long-term declarative memory: vast and persistent but requires retrieval. Episodic memory captures past experiences and sessions, while procedural memory stores learned skills and behavioral patterns. Effective agents combine all four rather than relying on any single layer.

The orchestration logic determines which memory types to consult for a given query — consulting all four on every request is wasteful. A lightweight query classifier routes to the appropriate stores: a factual question about the user's past triggers episodic retrieval, a domain knowledge question triggers external retrieval, a procedural question about how to format an output retrieves from the skill library. The assembled context from all relevant stores is then merged into the in-context memory for the current request. Latency budgeting matters: external and episodic retrieval add 50–150ms per request, which is acceptable for most agent applications but must be accounted for in SLA design.

Conversation memory is where most teams start because the problem is immediate and visible — users hate repeating themselves. Four strategies exist with progressively better quality-to-cost ratios: full history (naive), sliding window, summarization, and vector retrieval. Each solves the amnesia problem differently, and the right choice depends on conversation length, topic diversity, and whether sequential context matters more than relevance. Most production systems combine strategies — a sliding window of recent turns plus summarization of older turns plus vector retrieval of highly relevant past moments.

Summarization-based memory works by passing the oldest N turns through a fast, cheap model (GPT-4o-mini or Claude Haiku) with the prompt "summarize the key facts, decisions, and constraints from these conversation turns." The summary replaces the original turns in context, typically compressing 10 turns of ~800 tokens to a 150-token summary with minimal information loss for structured conversations. The compression ratio degrades for highly contextual conversations where exact phrasing matters — legal or medical contexts where precise wording is critical should use sliding window instead. Vector retrieval complements summarization by handling long-horizon references: if a user in turn 47 references a specific example from turn 5, semantic search retrieves it even if it was not captured in the summary.

Mem0 is an open-source memory layer for AI agents that abstracts the complexity of managing multiple memory stores behind a simple add/search API. Under the hood it combines LLM-based fact extraction, a vector database for semantic retrieval, and a graph database for relationship tracking — three separate systems coordinated by a unified orchestration layer. What makes Mem0 production-ready is its handling of memory updates and conflicts: when new information contradicts stored memory, it does not blindly overwrite — it versions the old memory and flags the contradiction for resolution. This versioning prevents the silent data corruption that plagues naive memory implementations.

The graph database layer is Mem0's most distinctive architectural choice. While vector search retrieves semantically similar memories, graph traversal retrieves structurally related ones — all memories about a specific entity, all facts connected by a relationship type.

Memories go stale. A user's preferred programming language, job title, or technology stack may change, and an agent that confidently personalizes responses based on an 18-month-old memory of "prefers Java" to a developer who has since moved to Rust creates the worst kind of experience — confidently wrong. Memory systems need explicit staleness handling: confidence decay over time, recency weighting in retrieval, and periodic re-confirmation for high-stakes persistent preferences. Forgetting, counterintuitively, is a feature rather than a failure — the right memory system knows what to discard.

Confidence decay should be implemented as an exponential decay function: confidence_t = confidence₀ × e^−λt where λ is the decay constant tuned per memory category. Facts (user's name, company) should decay very slowly (λ ≈ 0.001/day) while preferences (communication style, favorite tools) should decay faster (λ ≈ 0.01/day). The decay is halted and confidence reset to its original value on each reinforcing access — if a memory is retrieved and the user's behavior confirms it, confidence is maintained. Conflict resolution strategy should be configurable per memory category: for preferences, newer wins; for facts, higher-confidence wins; for constraints, both are preserved and flagged for human review.