Memory & Knowledge

Memory is the foundation AI Butler is built on. A context window is working memory: it fills up, and anything that only lives there eventually falls out. So AI Butler keeps long-term memory in a local SQLite database with FTS5 full-text search, a knowledge graph of extracted entities, vector embeddings for semantic search, and Reciprocal Rank Fusion for combined ranking. When you ask “what did I tell you about X last week?”, the agent is doing a real database query — not trying to compress your chat history through a language model.

This page is a hands-on tour. Every screenshot is live output from a running instance — no staging, no mocks.

The mental model

Think of AI Butler’s memory as a personal notebook with four spines:

Layer	What it holds	How it’s queried	Why it exists
1. Key Facts	Permanent facts extracted via rules: name, role, preferences, critical decisions	Injected into every prompt directly	Zero hallucination risk — no LLM summarization
2. FTS5 Full-Text	Every conversation turn, every saved note, every extracted fact	BM25 keyword search, sub-millisecond	Exact-match recall (“what did I say about Postgres?“)
3. Knowledge Graph	Entities (people, projects, decisions) + relationships	SQL joins over entity + relationship tables	Traversal (“who reports to Sarah?“)
4. Vector Embeddings	Semantic embeddings of every memory entry	Cosine similarity via pure-Go vec_distance_cosine function	Fuzzy recall (“anything about database trade-offs?”)

A hybrid searcher fuses the three retrieval layers (full-text, entity, vector) using Reciprocal Rank Fusion so a single query hits exact-match, entity lookup, and semantic similarity at once — Key Facts don’t need searching because they’re already in the prompt. Fusing on rank has a useful side effect: a memory surfaced by more than one layer collapses into a single result with a boosted score, so corroborated hits rank first instead of appearing twice. A gentle recency decay (90-day half-life, floored so old items always stay retrievable) nudges ties toward newer memories. You never pick which layer to query — the agent picks automatically based on what you asked.

Crucially: memory retrieval is search, not generation. AI Butler never asks an LLM to “summarize your chat history” or reconstruct facts from compressed text. What comes out of memory is exactly what went in, with a clear provenance chain back to the message that captured it.

Memory in action

The rest of this page is five live conversations that demonstrate each layer working for real. You can reproduce every screenshot on your own machine with a fresh install and the 3-command quickstart.

1. Save a real decision (capture + entity extraction)

Memory gets interesting the moment you ask the agent to remember something structured. Not just “remember that I like pizza” — a real decision, with reasoning and rejected alternatives:

You: Save this engineering decision to memory with full reasoning:

Decision: Use PostgreSQL instead of MongoDB for the user database in AI Butler.

Reasoning:

• Our queries are fundamentally relational (users → sessions → messages → memories)
• We need ACID guarantees for credential vault and capability audit logs
• SQLite FTS5 is production-grade for our full-text search needs
• We already have a pure-Go driver (ncruces/go-sqlite3) with zero CGO
• The single-binary promise precludes requiring a separate DB server

Trade-offs accepted: Harder horizontal scaling vs document stores. Schema migrations needed for every model change.

Store this as a proper decision in the knowledge graph, not just a note.

The agent calls memory.capture and memory.graph, extracts an EngineeringDecision entity with structured fields, builds relationships to the rejected alternatives, and catches a subtle inconsistency in the original prompt — the title said “PostgreSQL” but every bullet pointed to SQLite:

Look at the bottom of the response: "Discrepancy flagged & preserved — your decision heading said 'PostgreSQL instead of MongoDB' but every bullet in your reasoning points to SQLite. I stored the correct intent (SQLite) and noted the title mismatch in the record."

That is memory with integrity. The agent didn’t silently “fix” your input, didn’t discard the contradiction, and didn’t make something up to paper over the gap. It stored the reasoning, flagged the inconsistency, and left you in charge of the final answer. This is the posture every memory system should have, and it’s only possible because memory is a real database, not an LLM summarization.

2. Hybrid search in action (recall across sessions)

The real test of memory isn’t storage — it’s recall from a fresh session when you don’t remember the exact words. Open a new chat and ask an open-ended question:

You: I’m reviewing my architecture choices. Search your memory: what did I decide about the database, and why? Include both the decision and the reasoning behind it.

The agent calls memory.search (hybrid — FTS5 + graph + vector + RRF), finds the stored decision from a completely different session, and reconstructs every pillar of the original reasoning plus the rejected alternatives plus a closing meta-observation about the reasoning quality:

The user didn’t write “PostgreSQL” or “MongoDB” in the new question — they just said “the database.” The hybrid searcher matched via:

• FTS5 found the word “database” and surrounding tokens
• Knowledge graph traversed EngineeringDecision → appliesTo → AI Butler and EngineeringDecision → rejectedAlternative → MongoDB
• Vector embeddings matched “architecture choices” semantically to “single-binary promise”, “ACID guarantees”, etc.

Reciprocal Rank Fusion combined all three rankings and the top result was the original decision. The agent then rendered it back as a full 5-pillar explanation. This exact workflow — store a complex decision in session A, recall it with fuzzy natural language in session B — is the loop that makes memory real: nothing about it depends on the two sessions sharing a context window.

3. Build a real knowledge graph (people + relationships)

Memory isn’t just about facts — it’s about how facts connect. Tell the agent about your team:

You: Save these facts about my team to memory and build up the relationships:

• Sarah Chen is my product manager; she has final say on feature scope and priorities
• Marcus Rivera is my tech lead; he owns architecture decisions and mentors junior devs
• Jamie Kim is a backend engineer; she reports to Marcus and focuses on the agent loop
• Alex Patel is a frontend engineer; she reports to Sarah and owns the web chat UI
• Sarah and Marcus have weekly 1:1s on Monday mornings to align product and tech
• The team uses GitHub for code review and Linear for task tracking

The agent calls memory.capture for each fact and memory.graph to register entities and edges. Four Person entities (Sarah, Marcus, Jamie, Alex) are created with roles and focus areas; a Team entity ties them together; edges record the reporting structure (Jamie → reportsTo → Marcus, Alex → reportsTo → Sarah), the mentorship relationship (Marcus → mentors → Jamie), the UI ownership (Alex → owns → WebChatUI), and the weekly alignment ritual (Sarah ↔ 1on1 ↔ Marcus).

Everything is queryable not by text, but by graph traversal.

4. Query the graph (relationship-aware answers)

Now ask a question that only a real graph can answer cleanly — a question the agent has to walk edges to solve, not pattern-match against text:

You: Based on your memory, who on my team makes decisions about feature scope, and who should review code changes that affect the web chat UI? Use your knowledge graph to answer, don’t guess.

The agent calls memory.graph and memory.people, traverses the relationships, and returns two structured answers with explicit graph paths:

Look at the closing line: "Both answers come directly from the knowledge graph — no inference needed." The agent isn’t guessing. It isn’t pattern-matching. It walked the hasAuthority and owns edges and returned structured answers with full provenance. If you asked the same question to a memoryless LLM, it would hallucinate plausibly-wrong names. AI Butler queries its own data.

5. Memory stats + self-auditing

Memory hygiene matters over time. After you’ve captured a few hundred facts, how do you know what’s actually in there? The agent has tools for that:

You: Give me a full status report on your memory system right now:

• How many facts have I captured?
• How many entities are in the knowledge graph?
• How many relationships between entities?
• How many sessions have been indexed?
• What are the top 3 most frequently referenced entities?

The agent calls memory.stats and memory.graph, returns the numbers, and proactively audits the memory for integrity issues:

The agent flagged four real quality issues:

1. Duplicate identity — SL and Sam Lee are stored as separate Person entities; they should be merged
2. Stale preference — an old entry says Go is the user’s favorite language, but a newer entry says Rust; they conflict
3. Orphan node — the Project Apollo project has one observation and zero relationships, probably abandoned
4. Missing embeddings — zero vector embeddings stored, which means semantic search is currently keyword-only (works, but missing a layer)

And then offered to fix them. This is memory with agency, not just a passive data store. The same tools you use to query memory can also audit it, merge duplicates, and repair orphans. It’s the database treating itself like a first-class entity.

What just got demonstrated

Layer	Example above	Tool called
Key Facts	— always active, injected into prompts	(implicit in prompt composition)
FTS5 full-text	Example 2 matched “database” against the stored decision via BM25	memory.search (fts component)
Knowledge graph	Examples 1, 3, 4 — entity creation, edge traversal, path-based answers	memory.graph, memory.people, memory.facts
Vector embeddings	Example 2 matched “architecture choices” semantically to the decision	memory.search (vector component)
Hybrid ranking	Example 2’s answer came from RRF fusing all three rank lists	memory.search (top-level entry point)
Integrity & audit	Examples 1, 5 — discrepancy detection and orphan/duplicate flagging	memory.stats + LLM reasoning over the raw graph

Memory tools reference

Every tool is capability-gated — reads under memory.read, writes under memory.write, and permanent deletion under its own memory.forget capability so deployments can gate it more strictly — and logs to the audit trail. The agent picks which to call based on your natural-language question — you never name them explicitly unless you want to:

Tool	What it does	Capability
memory.capture	Save a fact, note, or decision with entity extraction	memory.write
memory.search	Hybrid search (FTS + entity + vector + RRF) across all memory	memory.read
memory.fts_search	Keyword-only FTS5 search for exact-match queries	memory.read
memory.thoughts	List captured thoughts filtered by tag or substring	memory.read
memory.facts	List stored key facts, optionally filtered by category	memory.read
memory.people	Query known people and their relationships	memory.read
memory.decisions	List tracked engineering/product decisions with reasoning	memory.read
memory.projects	List tracked projects and their status	memory.read
memory.graph	Traverse the knowledge graph by entity or relationship type	memory.read
memory.stats	Return counts, top entities, and health diagnostics	memory.read
memory.forget	Permanently delete a memory item, cascading to everything derived from it	memory.forget

Fact quality & correction (beta)

Facts aren’t flat strings — each one carries weight and provenance:

• Importance (1–10) — how much the fact matters, used by core-memory selection below.
• Confidence — a prior based on how the fact entered the system: stated by you in your own words (0.75), inferred from tool output (0.6), or explicitly corrected by you (1.0, definitive). Independently re-stating the same fact bumps its confidence.
• Provenance — a link back to the thought or transcript the fact was extracted from, so forgetting the source forgets everything derived from it.

Conflicts supersede, never silently overwrite. Single-valued facts (your name, your city, your current employer) carry a canonical key. When a new fact arrives for a key another active fact already holds, the old fact is marked superseded — kept, never deleted — the new fact becomes the single active holder, and the contradiction is recorded in a conflict ledger: auto_supersede when the newer statement is at least as trustworthy, needs_review when it’s markedly less confident. The invariant after every save: at most one active fact per key.

Conflicts are reviewable and reversible. The web dashboard’s Memories panel lists pending conflicts and lets you mark them reviewed or restore the old value if the replacement was wrong. The same panel exposes explicit correct, forget, and pin actions on any fact — a correction becomes the active fact with definitive confidence, superseding every other holder of the key. All of it mirrors into the audit trail. The opt-in reflection cycle also surfaces conflicts awaiting review in its nightly report so they don’t rot in the queue.

Forgetting is true deletion with cascade. The memory.forget tool permanently deletes a fact — or a thought/transcript along with every fact, embedding, and search-index entry derived from it — in one transaction. It’s registered under its own capability precisely because it cannot be undone.

Scored core memory (beta)

The always-in-context fact set (Layer 1 above) is chosen by a deterministic score computed in Go — no model call:

score = 0.40 × importance + 0.35 × recency + 0.25 × usage frequency

Pinned facts get an absolute offset no unpinned combination can reach — pin a fact and it’s always in context. Usage frequency is a real promotion signal: facts the model actually retrieves into turns earn their spot. The selected set is bounded by a token budget so core memory can never crowd out the conversation:

options:
  prompts:
    core_memory_selection: scored   # "scored" (default) | "recency" (legacy newest-first)
    max_core_memory_tokens: 350     # default

Memory banks (beta)

Memory banks are per-profile isolation namespaces. Every row — facts, thoughts, transcripts, entities, embeddings — belongs to a bank; the primary user’s bank is main (all rows created before banks existed belong to it). Background workers run against their own bank by default: a swarm subtask, for example, reads and writes an isolated worker bank, so nothing it captures, extracts, or embeds leaks into your primary memory, and retrieval never crosses banks unless a caller was explicitly handed another bank’s scope. As a hardening bonus, learned-instruction auto-detection only runs in the default bank — a background worker processing untrusted input can’t install a global instruction.

Context compaction (when conversations get long)

When a single conversation grows past the model’s context window, AI Butler compacts older messages algorithmically into structured summaries — preserving tools used, key files touched, pending work, and a chronological timeline. No LLM is used for the compaction, so there’s zero cost and zero hallucination risk.

The compacted summary is inserted as a system message at the top of the next turn, and the original messages are kept in the session transcript store (and thus in FTS5, the graph, and the vector index). You lose nothing — compaction is a rendering optimization for the model’s context, not a data deletion.

Properties you can hold it to

The memory system’s design goals, stated as checkable properties:

Property	How AI Butler delivers it
Local-first	One SQLite file (~/.aibutler/aibutler.db) on your machine — no memory leaves it unless you set up remote backup
Retrieval is search, not generation	Real database queries (FTS + graph + vector); what comes out is exactly what went in
Provenance	Every fact links back to the message that captured it
Correctable	Conflicts supersede with a review queue; corrections are definitive; restore is one action away (beta)
Truly forgettable	memory.forget deletes with full cascade — facts, embeddings, index entries
Multi-channel	Web, terminal, Telegram, Slack, Discord all share one store
Auditable	memory.stats, memory.graph, direct SQLite access, and an audit trail on every memory tool call
Exportable	cp ~/.aibutler/aibutler.db backup.db
Offline-capable	Works with Ollama; embeddings are optional — FTS + graph recall still works without them

Memory isn’t a feature bolted onto the side — it’s the architecture of the whole product. Every other feature — scheduling, channels, plugins, swarm, MCP tools — is built on top of it.

Architecture details (for the curious)

• SQLite via ncruces/go-sqlite3 — pure-Go WASM build, zero CGO, FTS5 included
• FTS5 uses BM25 ranking; the index is kept in sync with the source tables by SQL triggers on every insert, update, and delete
• Entity extraction is rule-based (regex patterns for common statements — no LLM calls, zero cost, zero hallucination risk); richer structured records like decisions come from the agent explicitly calling memory.capture and memory.graph
• Knowledge graph is two relational tables — entities and entity_relationships — traversed with SQL joins
• Vector search uses a pure-Go vec_distance_cosine function registered via SQLite’s custom function API; embeddings are stored as BLOBs in a memory_vectors table with UNIQUE(source_type, source_id) to prevent duplicates
• Embedding provider is auto-detected: Ollama’s nomic-embed-text if running locally, else OpenAI’s text-embedding-3-small if an API key is set, else embeddings are disabled and the system falls back to FTS + graph only (override via configurations.embedding)
• Reciprocal Rank Fusion combines rankings from the three search sources with the standard formula score(d) = Σ 1/(k + rank_i(d)) using k=60
• Recency decay multiplies each fused score by 0.5^(age/half-life) with a 90-day half-life, floored at 0.1 so old memories stay retrievable
• Memory banks scope every table by a bank column; the bank travels via context, so stores and tools need no per-call plumbing and unscoped code gets main

The full implementation lives in internal/memory/ — see fts/, entity/, graph/, vector/, hybrid/, and bank/.

Related

• Quick Start — see memory in 5 live scenarios
• Configuration Reference — memory + embedding settings
• Security Model — how memory is protected and audited