Agent Memory

Personhood through memory

A thought has been bugging me for years, and I have only just been able to put it into words. What are we without our memories? Are we the same person, or are we entirely new people? If I went to sleep tonight and woke up without memories, would I even miss them? Would I know that I am not myself?

This idea has haunted me ever since I watched Dollhouse. That show explores a technology that allows a person’s memory to be downloaded to a hard drive and replaced by someone else’s memories. The "doll" becomes a new person in the same body. In modern computing, there is a direct analogy. AI agents started with no memory at all except the context window, almost like Drew Barrymore’s character in 50 First Dates. They have their training data and anything in the context window, but when the window closes, they start fresh.

I have long dreamed of creating my own personal assistant. To do that, I realized that it needed real, persistent memory. I don’t mean memory that exists for a specific job; I mean memory that grows and never gets erased. This memory must also be simple enough to inspect and curate, and it must be private and portable. You could think of the agent harness (Claude Code, OpenCode, Pi) as the body and the model as the brain. If I control the memory, I can transport the "person" (as memory) to any "body" and use any "brain".

To make this a reality, I decided to start very simply. The simplest version was a Markdown file. Using a simple skill, I instructed the agent to keep track of things in that file. It was the agent’s memory and scratchpad. This worked well for a while; it would record and organize a task list, current focus list, and daily log. However, I realized that this approach would not scale. The next simplest solution was a local SQLite database using a single flat table with a jsonb field and a keyword field. Implementation was easy, and the skill was updated to state that the agent memory database is long-term memory, which can be queried for information and only appended to. It turns out that this was all I needed.

As the agent's memory grew, it began to do things that were more intuitive. It would remember what I meant when I said I was getting food for my wife. It knew who my kids were. It knew where I lived, and I didn’t have to tell it explicitly. This wasn’t creepy because the data sits on my computer in a database file that I can inspect, change, and back up.

With this setup, I can change models at will. I can change harnesses at will, and the memory remains the same.

Now, I know that there are more advanced memory storage systems out there. I haven’t upgraded to a vector plugin or anything more powerful. But this memory is fast, effective, and easy to understand. Sometimes, it’s best to settle for a "good-enough" solution so you can tackle harder problems.

Here’s a repo with my implementation. Feel free to download and try it out. Have fun!