Multiple timescales of information retention
Agent memory operates across three distinct timescales. Short-term or working memory is the current context: the conversation history, recent observations, and immediate state. This is typically stored in a context window (4K to 200K tokens for modern language models). Episodic memory records specific events: 'In the last conversation, the user asked X.' Semantic memory encodes generalizable knowledge: facts, procedures, world knowledge. A practical agent architecture separates these: context for current reasoning, a vector store for episodic retrieval, and the model's weights for semantic knowledge. This mirrors human memory, where you recall a specific conversation (episodic), remember that Paris is in France (semantic), and keep current task details in mind (working memory).
Building scalable memory systems
Working memory is limited by context window size. Beyond that, agents use episodic memory via embedding-based retrieval: convert queries and past events to vectors, find the most similar past events via nearest neighbor search, and include the top results in context. Semantic memory comes from retrieval-augmented generation: pre-compute embeddings of a knowledge base, then fetch relevant documents at query time. This three-level architecture scales to very long interaction histories and large knowledge bases. Modern implementations use vector databases (Pinecone, Weaviate, Chroma) for efficient nearest neighbor search. The architecture mirrors how humans use external tools (written notes for episodic, reference books for semantic) to overcome memory limits. Agents built on language models benefit greatly from this approach.