Memory System

The Moxxy memory system provides agents with persistent knowledge storage and semantic retrieval, backed by a single SQLite database and per-agent Markdown journals.

Overview

+-----------------------------------------------------+
|                   Memory System                      |
+-----------------------------------------------------+
|                                                      |
|  +------------------+    +----------------------+    |
|  | Long-Term Memory |    | Short-Term Memory    |    |
|  | (LTM)            |    | (STM)                |    |
|  |                  |    |                      |    |
|  | memory_index     |    | memory.stm_read      |    |
|  | memory_vec       |    | memory.stm_write     |    |
|  | (sqlite-vec)     |    | (session context)    |    |
|  +------------------+    +----------------------+    |
|                                                      |
|  +------------------+    +----------------------+    |
|  | Journals         |    | Compactor            |    |
|  | (Markdown)       |    | (Summarization)      |    |
|  |                  |    |                      |    |
|  | Append-only logs |    | Periodic journal     |    |
|  | Per-agent        |    | compression          |    |
|  +------------------+    +----------------------+    |
|                                                      |
+-----------------------------------------------------+

Storage Architecture

Single Database

All memory data is stored in the central ~/.moxxy/moxxy.db SQLite database using WAL (Write-Ahead Logging) mode. There is no per-agent memory.db file.

Key tables:

Table	Purpose
memory_index	Stores memory entries with metadata (agent_id, content, timestamps)
memory_vec	Vector embeddings for semantic search (via sqlite-vec)

Markdown Journals

Each agent maintains append-only Markdown journals in:

~/.moxxy/agents/{id}/memory/

Journals record interactions, decisions, and learnings as human-readable Markdown. They serve as a durable, inspectable log of agent activity.

sqlite-vec

The gateway registers the sqlite-vec extension at startup, enabling vector similarity search directly within SQLite. Memory entries are embedded and stored in the memory_vec table for semantic retrieval.

Memory Primitives

Agents interact with memory through four primitives:

memory.store

Stores information in long-term memory with a vector embedding for later semantic retrieval.

memory.store({
  "content": "The project uses PostgreSQL 16 with pgvector for production.",
  "tags": ["infrastructure", "database"]
})

memory.recall

Performs semantic search over long-term memory, returning the most relevant entries.

memory.recall({
  "query": "What database does the project use?",
  "limit": 5
})

The query is embedded and compared against stored vectors using cosine similarity.

memory.stm_read

Reads the current short-term memory (session context).

memory.stm_write

Writes to short-term memory for the current session.

Long-Term Memory (LTM)

LTM stores persistent knowledge that survives across sessions and agent restarts.

Characteristics

• Durable - Persists in SQLite until explicitly removed
• Semantic - Vector embeddings enable similarity-based retrieval
• Tagged - Entries can be tagged for categorization
• Per-agent - Each agent's memories are isolated by agent_id

How It Works

1. Agent calls memory.store with content
2. Gateway generates a vector embedding for the content
3. Content and embedding are written to memory_index and memory_vec tables
4. On memory.recall, the query is embedded and compared against stored vectors
5. Top-k most similar entries are returned to the agent

Short-Term Memory (STM)

STM holds the current session context -- recent messages and working state.

Characteristics

• Session-scoped - Tied to the current run/conversation
• Fast access - Kept in memory with database backing
• Automatically managed - The Executor maintains STM during the agent loop

Primitives

• memory.stm_read - Agent reads its current session context
• memory.stm_write - Agent writes notes or working state for the current session

Journals

Markdown journals provide a human-readable, append-only record of agent activity.

Location

~/.moxxy/agents/{id}/memory/

Format

Journals are plain Markdown files with timestamped entries:

## 2026-03-30 10:15:00

Researched transformer architectures. Found three recent papers on
efficient attention mechanisms. Stored key findings in LTM.

## 2026-03-30 11:30:00

User asked about database schema design. Provided recommendations
for the user's PostgreSQL setup based on their requirements.

Inspection

Journals can be read directly with any text editor or viewed through the agent's memory primitives. They serve as an audit trail of what the agent has done and learned.

Compactor

The compactor is a background process that periodically summarizes journal entries. As journals grow, the compactor:

1. Reads older journal entries
2. Generates summaries using the configured LLM
3. Stores summaries as condensed LTM entries
4. Keeps journals manageable in size

This ensures that agents retain important knowledge from long histories without unbounded journal growth.

Memory in the Agent Loop

During execution, the Executor automatically manages memory:

1. Context building - STM and relevant LTM entries are injected into the LLM prompt
2. Semantic retrieval - When the agent's prompt touches on prior knowledge, relevant memories are retrieved
3. Memory updates - The agent can explicitly store new information via memory.store
4. Journal appending - Interactions are logged to the agent's Markdown journal

API Access

Memory can also be queried through the REST API, though the primary interface is through the agent primitives.

Best Practices

What to Store in LTM

Good candidates:

• Important facts and decisions
• User preferences and context
• Project-specific knowledge
• Recurring patterns and solutions

Avoid storing:

• Temporary calculations
• Verbatim conversation transcripts
• Ephemeral status information

Memory Hygiene

• Let the compactor handle journal growth automatically
• Use tags in memory.store for better organization
• Use specific queries in memory.recall for better retrieval
• Reset sessions when switching to unrelated tasks

Troubleshooting

Semantic Search Returning Poor Results

1. Ensure sqlite-vec is loaded (check gateway startup logs)
2. Verify that entries were stored with memory.store (not just written to journals)
3. Use more specific recall queries

Memory Not Persisting

1. Check that the gateway is running and moxxy.db is writable
2. Verify the agent is using memory.store for persistent data
3. STM is session-scoped and will not persist across restarts -- use LTM for durable storage

Database Size Growing

The SQLite database includes both memory entries and vector embeddings. To reclaim space:

sqlite3 ~/.moxxy/moxxy.db "VACUUM;"