The Memory Problem I Actually Solved

Updated: Oct 30
Here is a conversation I had with AI chatbots more times than I can count.
I would spend an hour explaining my business — the products, the people, the plan we had just agreed on. The chatbot would nod along, repeat it back perfectly, and I would think: finally, it gets it.
Then I would come back the next day, and it was gone. Not fuzzy. Gone.
I was re-explaining my own business to a machine that had sworn up and down it understood me, and starting over from an empty room.
If you run a business by yourself, you do not have time for that. Your memory is already carrying the customers, the bills, the family, and the forty open loops in between.
A tool that forgets is not an assistant. It is one more thing to manage.
So I stopped asking how to get a chatbot with a bigger memory. I built something else instead: a second brain for my business — a system where the memory does not live inside the chatbot at all.
Let me be clear about what I ended up building, because the name sounds familiar and the thing itself is not.
People talk about a "second brain" all over the AI industry right now. Most of what they mean is a notes app, a chat history, or a plug-in that remembers a few facts about you between sessions.
Mine is none of those.
This is a much improved second brain over what most of the AI industry has — checked before it is filed as true, governed by a rule for what earns the right to stay, built to run a whole business, and — as you will see further down — built for a whole crew of agents, not just one assistant.
That is a big claim. The rest of this article is me showing my work.
A desk is not a filing cabinet
Here is the plain-words version of what is wrong with ordinary AI chat.
Everything a chatbot "knows" during your conversation sits on its desk — engineers call that desk the context window. While your papers are on the desk, the chatbot is brilliant with them.
But desks get cleared. The session ends, the desk is wiped, and tomorrow your brilliant assistant walks into an empty office with no idea who you are.
Now, the industry's big answer to this is: build a bigger desk. A million-token desk. It sounds impressive, and it is still a desk. Bigger, yes. Cleared just the same.
My system takes the opposite approach:
Do not make the desk bigger. Build the office around it — a filing cabinet, a card catalog, and someone whose job is to keep the files straight.
The memory of my business — its second brain — lives in the office, not on any one desk.
Any agent who walks in — smart, forgetful, brand new — can be handed the right file at the right moment and work like they have been here for years.
My second brain has three parts
I run a one-person business with a small crew of AI agents, and everything they know together lives in three places:
1. The card catalog. A master index that says what exists, what is current, and what has been replaced.
Think of the old library card catalog: it does not hold the books, it tells you exactly where every book is and which edition is the real one. When two files disagree, the catalog — not somebody's guess — settles it.
2. The filing cabinet. The actual documents: research, plans, decisions, the full text of what we learned and when we learned it.
Nothing important lives only in a chat window. If it matters, it gets filed.
3. The desk notes — the second brain proper. A distilled layer of checked, settled knowledge — the short version an agent can absorb in seconds before it starts work.
Not rumors, not half-finished drafts. The verified stuff.
Here is the part that makes it work: a piece of information does not get promoted into the second brain just because somebody said it. It gets checked against its source first.
Numbers get matched exactly. Dates get stamped. If a claim cannot be verified, it is flagged as unverified — never quietly dropped, and never quietly believed either.
That one habit — check before you file it as true — is most of the difference between a system that remembers and a system that confidently misremembers.
This is not the industry's version of a second brain
I want this part unmistakable, because the industry is selling a lot of things under this name right now.
Most AI memory extracts a few facts from your chats and plays them back next time. Mine is the system of record for a whole business — decisions, prices, rulings, people, projects, and preferences — checked against their sources, not just scraped from conversation.
Most AI memory belongs to one assistant, one app, or one model company. Mine lives outside all of them, in the office around the desk. Change the model, change the vendor, bring in a brand-new agent — the brain is still there, unchanged.
Most AI memory keeps everything, so the pile grows until nothing in it can be trusted or found. Mine has a rule for what earns the right to stay — that rule is the next section, and it is why this brain gets sharper over time instead of just bigger.
Most AI memory remembers whatever was said, right or wrong. Mine checks before it files, flags what it cannot verify, and archives what has been replaced instead of letting an old fact quietly fight a new one.
Put those together and you do not have a slightly better chat memory. You have a different thing: a business memory that does not live and die inside somebody else's chatbot.
My favorite rule: a file earns its right to live
Most systems keep everything forever, until the pile is so big that nobody can find anything and the whole cabinet is useless.
Mine has a filing rule I am genuinely proud of. It came from watching what actually happens to information in a real business:
If something living depends on it, it never expires. A fact that one of my published articles, websites, or projects is built on stays current for as long as that thing is alive. The system will not let the foundation quietly rot out from under a finished piece of work.
If it was used and then replaced, it goes to the archive. Not deleted — archived. Out of the everyday cabinet, but retrievable forever. It earned that by having mattered once.
If it was never used at all, it gets buried. Every piece of research that comes in gets a shelf life. If nothing ever depends on it, when the clock runs out it leaves the active files. Honestly? Small celebration.
A memory that keeps everything remembers nothing well.
Burial for what never mattered. The archive for what mattered and was replaced. Permanence for what something living still stands on.
That is the whole rule, and it is why my second brain gets sharper over time instead of just bigger.
One small example, from a real week
A while back I settled a hardware question after a fair amount of research — which parts, at what checked prices, for which job. It was the kind of decision you make once and then rely on.
Weeks later, in a completely different conversation, a similar machine came up. The quick answer floating around was wrong in a way that would have cost me money: it quietly contradicted the decision I had already made, because the new conversation had never heard of the old one.
In an ordinary chat setup, that is just how it goes — the old decision lived in a dead session.
In my system, the decision was on file. The record got checked first, the settled answer surfaced, and the tempting new answer got measured against it instead of replacing it.
Nothing dramatic happened, and that is precisely the point: the most valuable thing a memory does is stop you from re-deciding, badly, something you already decided well.
Multiply that by every price, preference, ruling, and hard-won fact in a business, and you see why I stopped shopping for bigger desks.
One brain. A whole crew of agents reading it.
Here is the part that has grown the most since I first built this — and the part I have not seen anywhere else.
My second brain is now multi-agentic. It is not built for one agent. Many agents read from it, and many agents write back to it — the same checked brain, shared by the whole crew.
Start with the three agents I work with every day:
Altair, my Senior VP Agent. Altair is the one I chat with daily. Before Altair starts work, Altair reads the second brain. After we decide something, that ruling gets written back — so tomorrow's conversation, and everybody else's, starts from what we actually settled.
The CEO agent. Every operator who runs this system gets to name their own CEO agent — that is why I call it the nameable CEO agent. Mine works in the background on the longer, bigger jobs. It reads the same second brain Altair reads, so it never needs me — or Altair — to re-brief it on my own business.
Lyra. Lyra watches the health of the whole ecosystem — the machines, the systems, whether things that should be running are still running. Lyra reads the same brain too, so a health warning lands in the same context as the decisions and projects it affects, not in a separate silo nobody connects.
Then it goes wider than those three:
Any agent that can reach my ecosystem can be served by it. If an agent can Tailscale SSH into my ecosystem — under my rules and my permissions — it can be handed the same files, the same checked record. It does not matter which company made that agent, which model powers it, or whether it runs here at home or out in the cloud. The brain lives outside the agent, so a new or forgetful agent stops being a problem.
Every spawned subagent and worker reads and writes it too. When Altair, the CEO agent, or Lyra starts a job that is too big for one pass, they spawn subagents and workers to do pieces of it. Those workers are temporary — some live for minutes. In most systems, whatever a temporary worker learns dies when the worker does. In mine, the worker reads the second brain before it starts, and writes what it found, checked, or decided back into it before it finishes. Temporary agent. Permanent memory.
Think about how unusual that is.
In most AI setups today, every agent has its own little memory — or no memory at all. Put three agents on one business and you get three versions of the truth, drifting apart, and one very tired human in the middle carrying messages and facts between them. People call that "multi-agent." Mostly it is multi-forgetting, coordinated by you.
Even the better setups usually share only a temporary workspace — a scratchpad for one task, in one session, that evaporates when the job ends.
Mine is one permanent, verified, shared brain across agents, across models, across sessions, and across time. My agents can hand work to each other at 3 in the morning without me being the memory between them. When they disagree, they do not guess — they check the catalog, the same way you saw in the example above.
I used to be the only memory between my tools. Now the system is.
That, as much as anything in this article, is what I actually solved.
What this feels like from my chair
I am one person, running several businesses, and I sleep odd hours. My system works while I am away, and it works with agents that — on their own — would forget everything by morning.
The reason they do not is that none of them is trusted to be the memory. The office is the memory.
So decisions I made months ago show up in today's work without me re-explaining them. A fact I checked once stays checked.
And when an outside tool with a famously bad memory plugs into my system, it performs like it remembers — because the second brain remembers for it, and hands it the file when it matters.
That is the moment my users describe, and honestly the moment I still enjoy myself: the system recalls something a chatbot forgot four times, and it is not a bigger desk that did it.
It is a second brain with a filing cabinet under it — and the discipline to check things before you file them as true.
The honest part — and where this goes next
This is built, and it runs my business every day — but it is not finished, and I do not sell finished.
I am still improving how new information gets routed to the right file, and how the stores get reconciled against each other. I build in public, and the receipts — including the parts that break — are part of the record.
And I can already see the next two steps for the second brain itself, because the same problem it solved for me and my agents is sitting one level up, twice over:
Next: an ecosystem-wide second brain. Today, one second brain serves one ecosystem — my agents, my business, my machines. The next step is for that brain to span the multiple nodes of an ecosystem as one: the machines at home, a machine on the road, a server out on the internet — different places, different nodes, one shared, checked memory. A decision made on the road shows up at home. Work started on one node is understood on another, without a human carrying it across.
After that: a community second brain. Teams of people — and whole communities — who are working together or collaborating together should be able to share a second brain for that shared work: the same checked facts, the same decisions, the same project history, in front of everybody's agents at once. Your private brain stays private — nobody is proposing you hand your whole business to a group. But the work you are doing together gets one common, trusted record, instead of five people and fifteen chatbots all starting from five different versions of what was agreed. If you have ever watched a team re-litigate a decision that was already made in a meeting nobody's AI attended, you already know why this matters.
That is the direction: one person and a crew of agents today, an ecosystem of nodes next, and people and communities collaborating on a shared brain after that. Each step is the same idea, wider: stop trusting the desk, and build the office — then build offices that can work together.
SuccessBrian OS is free and open. The code is on GitHub, my data stays mine (you get the system, not my business), and the best way to understand any of this is to watch it run.
Get SuccessBrian OS on GitHub — free, Apache-2.0, bring one machine and a business that needs remembering.
Dig deeper
Four good videos on the same problem, from four different directions. Watch them, and notice what each approach keeps — and what it leaves out.
Building a Second Brain: Capturing, Organizing, and Sharing Knowledge Using Digital Notes — Tiago Forte
The talk that named the idea: a second brain for your notes — capture, organize, distill, express — built for one person's thinking. Mine runs the same instinct one level up: the second brain is not a notebook I keep for myself, it is the shared, checked memory a whole crew of agents works from.
An engineer's talk reaching the same diagnosis from the other side: a context window is not memory, and agents need a real memory layer underneath — formed, stored, retrieved, governed. That layer is what I built. The part his talk leaves open is the part I care about most: a rule for what memory should keep, and what it should let go.
A memory layer you plug into an app: it pulls facts out of conversations and hands them back next session, so the assistant stops starting from zero. Mine is not a plug-in — it is the system of record for a whole business: decisions, prices, and rulings, checked against their sources, archived when replaced, and buried when nothing ever used them.
A short, clear explainer of context engineering — the craft of managing what sits in the window, because a crowded desk makes for a worse answer. All true, and all still about the desk. My approach leaves the desk small on purpose and puts the memory in the office around it.
Read more from Brian Lathe on Substack
This blog is one room in a bigger house. My main writing lives on Substack - longer pieces, new posts first, always free to read: Brian Lathe on Substack.



Comments