Local-first and plain text: seven storage surfaces, one retrieval gate, and one contract, drawn as a single picture.
Companion to the essay Your AI Forgets Everything. Here Is the Plain-Text Fix I Built.
The essay explains why I built my agents a memory and how I decided whether it earns its keep. This is the same system seen from above: every surface it stores to, the one gate it reads through, and the contract that keeps three agents from overwriting each other. Everything here is plain markdown on my own machines. Nothing runs on a paid memory service, and none of it is hard to rebuild. Read the diagram first, then use the sections below as the reference.
Every session runs the same loop. An agent reads down: the startup surface and index are already loaded, and the gate reaches deeper only when it has to. When the work is done, it writes back up: summaries, handoffs, and an updated startup surface, every write passing through the contract. The human sits to the side, curating what graduates into the wiki and reading it back, never inside the agent loop.
Teal traces the read path, cheapest tier first. Gold traces the write-back, committed through the contract. Dashed is the human side: you curate and you read. The numbers on each box match the surface cards below.
Claude Code
Primary builder, architect, and reviewer. Owns ~/.claude/.
Codex
The second coding agent, working its own lane of projects. Owns ~/.codex/memories/.
Hermes
A conversational second brain on call over Telegram, with its own native memory.
Subagents
Scoped, single-task agents dispatched on demand and retired when the task is done.
1. Startup surface
working-memory.md
What is active right now: hot, warm, and archive, each with its exact next action. A whiteboard, kept brutally short so the agent orients in seconds.
2. Index
MEMORY.md
One line per durable note, loaded every session. It is a router: the agent does not need to hold a fact, only to know which file holds it.
3. Durable memories
memory/*.md · one fact per file
Typed as preference, decision, project, or reference. Atemporal facts that stay put, each in its own file so any one is cheap to grep, prune, or delete.
4. Conversation summaries
memory/conversations/
The episodic layer: time-stamped checkpoints of what got built, verified, and broke, plus the next step. This is how I reconstruct a decision six weeks later.
5. Project handoffs
per-project dirs · *HANDOFF* / .codex-save.md
Resume state that lives with the project, not the central store: what is done, what is blocked, what not to touch, and the exact next step.
6. Shared long-term store
ShadowMind Memory/
Cross-agent decisions and the contract itself, readable by every agent. The constitution, not the daily news.
7. Human wiki
Obsidian · PJ Vault on SeaDrive
Curated and selective, for me rather than the agent. Kept separate from agent memory on purpose, so neither turns into a swamp.
Most memory guides say search before you answer. Searching often costs more than reading the file would have, so retrieval runs through a gate instead. It reaches for the cheapest tier that answers the question and stops there.