Memory Architectures & Context Pruning

Context windows may reach 1M+ tokens, but “lost-in-the-middle” attention degradation and linear cost scaling make context hygiene critical in production agents.

The Three Tiers of Agent Memory

Production systems decouple memory into three specialized tiers:

┌────────────────────────────────────────────────────────┐
│ 1. Working Memory (Active Loop Context)                │
│    - Recent 5-10 turns, immediate tool returns         │
├────────────────────────────────────────────────────────┤
│ 2. Semantic State (Relational / KV Database)           │
│    - Verified variables: user_id, active_plan, credits│
├────────────────────────────────────────────────────────┤
│ 3. Episodic Memory (Vector Index / Summary Archive)    │
│    - RAG over historical past sessions                 │
└────────────────────────────────────────────────────────┘

Sliding Windows vs Incremental Summarization

When message history exceeds 20 turns:

Sliding Window: Discard the oldest messages, keeping only the system prompt and the latest $N$ user-assistant pairs.
Summary Compression: Trigger a background LLM call to condense the earliest 10 turns into a structured markdown block (<previous_context>...</previous_context>).

Truncating Tool Payloads

The number one cause of context window overflow is raw tool output. Always prune server responses before returning them to the model:

def format_tool_result(raw_records: list[dict]) -> str:
    # Filter to only the 4 fields the agent actually needs
    condensed = [
        {"id": r["id"], "name": r["name"], "status": r["status"]}
        for r in raw_records[:10]
    ]
    return json.dumps(condensed)

Continue to Module 3: Designing Bounded Agents.