Persistence of Memory for Smart Websites | Steve Robison

Most of what we call "AI" today is still object-level cognition with amnesia.

A model answers. A tool runs. A context window fills and empties. Tomorrow the system wakes up brilliant and blank -- no durable sense of what it knows, what it tried, what failed, or why a strategy worked last time. That is not intelligence in any serious sense. It is performance without self-regulation.

Human minds are not only capable. They are metacognitive.

In the late 1970s, developmental psychologist John H. Flavell formalized metacognition as knowledge and cognition about cognitive phenomena -- the active monitoring and consequent regulation of thinking itself. "Thinking about thinking." Not a slogan. An architecture of mind.

Flavell's model still maps cleanly onto what serious AI systems will need:

Metacognitive knowledge -- beliefs about persons (self/other as thinkers), tasks, and strategies; declarative, procedural, and conditional ("what / how / when and why").
Metacognitive experiences -- confidence, doubt, feeling-of-knowing, sensed difficulty.
Goals / tasks -- what the activity is for.
Actions / strategies -- what is actually done.

Around those sits the regulation loop: plan → monitor → evaluate → control. A student notices they are not retaining a chapter and switches to active recall. A driver feels fatigue and pulls over. A writer senses a paragraph that will not flow and revises. Intelligence is not only inference. It is adaptive self-regulation.

That is the scientific center of what I am building -- and I am building it as working software, not a slide deck.

From psychology to machine architecture

If metacognition is real in humans, then "real intelligence" in artificial systems will not arrive by scale alone. Scaling improves object-level fluency. Metacognition requires a meta-layer: something that monitors, evaluates, and controls the primary reasoning and memory processes.

Three engineering primitives make that tractable:

Persistent memory as metamemory

Long-term stores -- vector memory, structured metadata, episodic logs, archival lakes -- are the machine analogue of human long-term memory. Metamemory is what turns a dump into a mind:

Know what you know (confidence, source, access history).
Decide what deserves persistence versus transient handling.
Consolidate: abstract, link, summarize.
Forget strategically -- prune noise without erasing signal.
Retrieve with awareness of relevance, reliability, recency, and gaps.
Detect contradiction and incompleteness, then act.

Gating as metacognitive control

Gates are not only learned filters. At the meta-level they become dynamic policy: open or close write/read paths based on uncertainty, predicted utility, novelty, goal alignment, consistency, salience. High-value or uncertain-but-promising material gets rich metadata and strong write priority. Redundant noise is compressed or refused. Memory bloat is an intelligence failure, not a storage problem.

Recursion as self-reference

The system must store not only facts, but records about its own processes: why this was kept; what confidence accompanied it; which retrieval strategy worked in similar cases. Reflection outputs become new experiences. Hierarchies form: raw events → episodes → themes → meta-strategies. That is how memory management itself improves -- recursive self-improvement without mystical hand-waving.

How it is implemented (the stack)

Vision without systems is poetry. Systems without vision are plumbing. I want both.

Persistence of Memory is a hybrid memory plane I designed and operate end to end:

Semantic / vector store -- synapses: embed, search, retrieve by meaning -- LanceDB, local embedding models, dense vector encodings
Structured state -- queues, pipeline status, orchestration metadata -- MySQL (schema, indexes, app users -- DBA work, not an afterthought)
Long-term archive -- partitioned event history -- JSON data lake
Memory API -- ingest, status, table search/insert -- Python FastAPI web service (Bearer auth, health checks, LaunchAgent/systemd)
Orchestrator & dashboard -- coordination, visibility, higher-level logic -- PHP (Slim), nginx, PHP-FPM
Object-level cognition -- reasoning, content actions, reports -- Grok (xAI), Claude, local Ollama where appropriate
Build loop -- architecture, implementation, iteration -- Cursor as the agentic IDE, with human judgment on the constitution and the gates

Write path (curated): site or agent → POST /events/ingest → queue → optional LLM consolidate → embed → LanceDB synapse, with MySQL tracking the pipeline. Not every log line earns a vector. Selection at the door is metacognitive gating in code.

Read path: POST /tables/{pool}/search with query_text → nearest neighbors in that pool's vector space → compact recall injected into the next prompt cycle.

Clients: thin PHP MemoryClient patterns over HTTPS -- curl, JSON, tokens -- so production sites talk to the memory plane as ordinary web services, not magic.

Isolation: purpose-scoped LanceDB tables / pools (source_db + table name). One engine. Many lineages. Constitutions stamp goal constraints onto deposits and prompts.

I run the ops side too: local TLS hosts, nginx vhosts, MySQL bootstrap, daemonized LanceDB, cron at the edge, batch sync when live cross-host calls are the wrong honesty. Architect, sysadmin, DBA, API designer, and coder -- because metacognition that you cannot deploy, observe, and repair is not a system. It is a wish.

Two layers, one loop

Object level: Grok/Claude/Ollama + tools + working context; proposed memory operations.
nMeta level (emerging): monitoring, reflection, metamemory strategy, gating -- today partly in constitutions, typed deposits, curated ingest, and human-visible synapses; tomorrow a clearer controller module.

Flow:

Propose → evaluate (utility, confidence, consistency, goal relevance) → gate → act → log signals → update meta-knowledge → reflect when stakes or novelty justify the cost.

Bounded depth. Termination criteria. Lightweight monitoring by default; deeper reflection when metacognitively warranted.

This is not a chatbot feature. It is an attempt at self-regulating intelligence: continuity across sessions, calibration against overconfidence, coherence that survives context reset -- implemented with real databases, real APIs, and real failure modes you can tail in a log.

Why constitutions and pools matter

Metacognition without a goal model is clever drift.

A constitution is the stamped north star -- voice, boundaries, success criteria -- so monitoring has something honest to monitor against. Separate memory pools are purpose-scoped metamemory. That is how you avoid the Dunning-Kruger of agents: perfect confidence about the wrong objective.

What I believe

I do not think the next leap is a louder demo. I think it is systems that can know that they know, sense when they do not, regulate what they store, and improve the strategies by which they think -- on infrastructure you can own, query, and govern.

Passive archives are not minds.
nStateless fluency is not wisdom.
nScale without metacognition is a brighter amnesia.
nPrompts without architecture are cosplay.

Persistence of Memory is the hypothesis made runnable: LanceDB vectors + MySQL truth + Python services + PHP orchestration + multi-model cognition (Grok, Claude, local LLMs) + Cursor-accelerated engineering -- aimed at something closer to savvy: coherence, self-understanding, and participation that compounds.

Progress, not perfection.
nOne careful loop at a time.

If you work at the intersection of cognitive psychology and agent memory -- calibration, metamemory, reflection architectures, gated persistence, or the unglamorous craft of making it all stay up -- I would genuinely enjoy comparing notes.

-- Steve