RFC-0002: RMS Event Store and Append-Only Log

Status: Draft Tier: 1 — Storage and Integrity Version: 1.0 Date: 2026-02-19 Depends on: RFC-0000, RFC-0001

1. Abstract

This RFC specifies the Event Store: the append-only, tamper-evident log that is the single source of truth for the entire RMS system. It defines the storage schema, operation types, integrity guarantees, compaction semantics, and the portability artifact format.

All derived state in RMS — memory objects, graph edges, entity snapshots, scores — is computed from this log. If derived state is lost or corrupted, it is rebuilt by replaying the log. The log itself is never rebuilt; it is always the ground truth.

2. Storage Backend Requirements

Implementations MUST use a storage backend that provides:

• Append-only writes (no in-place modification of committed records).
• ACID transaction semantics for individual write operations.

PRME graph replacements use a transaction covering lifecycle state, replacement pointer, provenance edge, and a checksummed SUPERSEDENCE_APPLIED operation with complete before/after snapshots. Its deterministic pair identity makes exact actor/evidence retries durable and rejects conflicting retries. supersede_many applies a batch atomically and rejects self-replacement, retired replacements, or pairs across users/scopes. New LLM ingestion uses the journaled derivation protocol in RFC-0016: saved extraction and prepared inputs, idempotent index staging, and fenced atomic graph publication. New duplicate/alias merges atomically append a checksummed ORGANIZER_MERGED operation containing complete node and relationship inputs and outputs. New unverified alias links similarly use deterministic pair and edge identities and commit a checksummed ALIAS_PROPOSED record with complete node inputs and the published edge. This is distinct from full replay of historical organizer and manual mutations, whose complete operation inputs are not all journaled. TTL expiration also uses a backend transaction to bind the archived graph state to a deterministic, checksummed TOMBSTONE_SWEEP record. Its portable summary contains the RFC-0007 policy, reason, content hash and clocks; its authoritative record retains complete before/after nodes. External index eviction follows the durable transition.

• Efficient range scans by timestamp and stream.
• Content-addressed deduplication by content_hash.

The reference implementation uses DuckDB for the event store. Implementations MAY use alternatives (SQLite, PostgreSQL, a custom log format) provided the above requirements are met and the portability artifact format (Section 9) is supported.

Local engines optionally accept duckdb_threads at instance creation. The default retains DuckDB’s worker setting; conflicting settings for concurrent opens of the same file fail before schema initialization. This runtime resource control does not change event semantics or persist in the portable pack.

An optional expected namespace_id binds a fresh local pack inside DuckDB and must match on later opens, before normal schema/backfill/index startup. Existing unbound packs are not silently adopted. The workspace registry records initialized projects so a missing database cannot be replaced silently. This identity metadata is part of the physical artifact, not a shared-table filter or access grant. PostgreSQL workspaces similarly bind schema identity before engine startup and publish initial schema/registry state in one transaction. Missing previously initialized relations cannot be silently recreated. PostgreSQL backup/restore must preserve both the registry and project schemas; see workspaces.

Raw-source recovery

Both fast ingestion and LLM ingestion atomically queue raw-source indexing with the event. A provider failure leaves that job pending; scoped retrieval or explicit processing can materialize the original NOTE after restart. Its ID and timestamps come from the source event. Processing completion acknowledges this raw index, not LLM derivation completion. Model summaries cannot overwrite source indexes. Events are immutable, so the API’s inherited updated_at equals created_at rather than the time a row happened to be read.

ingest_fast_many() extends that boundary to an ordered, single-owner raw batch. All items and metadata snapshots are validated before acquiring a backend connection. DuckDB and PostgreSQL commit every event and materialization job in one transaction or roll back the whole batch. Graph and index work remains deferred, owner scoped, bounded, and restart safe; event IDs preserve input order. This path creates deterministic raw NOTEs and does not accept typed-node overrides or queue model extraction. An optional caller request UUID derives an owner-scoped operation identity. Its checksummed input binding, ordered event IDs, immutable events and materialization jobs commit in the same transaction. An exact retry returns the retained IDs after restart; changed inputs fail, and an unkeyed call remains a separate admission. PostgreSQL arbitrates concurrent same-key requests at the operation insert. DuckDB may surface a native transaction conflict across engine instances; a fresh retry resolves the saved record without duplicating events.

Historical source clocks

LLM ingestion accepts an explicit timezone-aware event_time through the engine, synchronous client, HTTP and MCP. Batch messages can supply individual clocks. timestamp continues to record admission time. Relative temporal references in new derivation plans use event_time when provided and otherwise timestamp. Raw NOTE recovery retains the saved source clock, and extracted claims without a resolved temporal reference inherit it. Explicit older-effective replacements remain historical rather than retiring later facts. Existing saved plans retain their original timestamps on retry; this is not a retroactive temporal migration. New extracted facts set valid_from to that resolved source-effective time. Direct Python, HTTP, and MCP storage can provide a separate timezone-aware valid_from and exclusive valid_to; omission retains the admission-time start. An end requires an explicit earlier start, and invalid intervals fail before the event or recovery job is admitted. New source-grounded replacements close the prior validity interval inside the derivation commit when the boundary is not earlier than its stored start. Older ingestion-based intervals that cannot be closed without inversion retain their stored dates and are retired by lifecycle and supersedence state. Timezone-free new imports are rejected; missing source time is not guessed. This admission check also applies to Python store() and ingest_fast() and their synchronous client equivalents, before any event or recovery job is written. MCP memory_store accepts the same aware source clock; returned nodes expose event_time, valid_from and valid_to separately. An omitted source clock remains null through raw-source and direct-store recovery. Existing stored rows are still readable under their original semantics.

Direct typed storage recovery

New event and direct-node metadata are validated and copied to a finite JSON snapshot before waiting for the backend lock/connection. Non-finite or unsupported values and object keys that collide during JSON normalization fail before event/work admission on both backends. A string key and a numeric key must not silently overwrite each other after serialization. Existing source rows are not rewritten. Legacy non-finite graph metadata can be retained in lifecycle, reinforcement and merge journals using the versioned prme-special-floats-v1 path encoding; finite snapshot bytes and old raw checksums remain unchanged. See metadata handling for encoding and compatibility boundaries.

Content hashing covers the exact source string, including empty text and whitespace. New empty events carry SHA-256 of the empty byte sequence, not an empty hash field, so they can satisfy the same durable source-binding checks. This correction does not rewrite historical event rows.

New store() calls commit the event, an event_materializations job and a DIRECT_STORE_REQUESTED operation in one database transaction. The operation contains a versioned complete initial MemoryNode snapshot and source binding, including its generated ID, classification, confidence, timestamps and TTL. Its UUID is derived from the event ID. A checksum covers the serialized record; the record remains a string within the JSON payload so JSONB numeric normalization does not change it. Reads verify the checksum and source owner, scope, session, content hash and evidence reference before recovery. Version 1 requires event and node content to match and retains its original bytes and checksum. Version 2 allows an explicit retrieval_content projection: the exact caller source stays in the immutable event while the separately checksummed node text is indexed and packed into model context. Version 2 rejects an identical projection so ordinary stores continue to produce version-1 records.

Direct stores may also attach versioned typed value bindings in event and node metadata. Each caller-supplied presentation form must occur verbatim in both the source and indexed text. Retrieval exposes a binding only when that form survives packing; exact complete tool-argument resolution returns source-node provenance and rejects conflicting mappings. This operational mapping is not model inference, semantic validation, substring rewriting or generated-answer repair. It uses the existing direct-store record version because the complete metadata snapshot was already part of both versioned node formats.

Recovery creates a missing graph node from the saved values and repairs its indexes without an LLM. Existing graph state is retained, including retirement. Lexical replacement commits deletion and insertion together; vector replacement publishes a new durable vector before removing old keys. Failure retains pending work and the previously healthy search path. processing_status() completion covers this node and indexing, not optional reinforcement, supersedence or QA pairing. A graph creation failure raises MaterializationError with the accepted event ID; index failures remain nonfatal after confirming the node is durable. Legacy direct stores have no repair record and are not retroactively queued. This does not add cross-process work fencing or exactly-once execution.

The unreplayed QA-pair heuristic is disabled by default. Explicitly enabling it is a hypothesis for applications willing to accept an in-process, best-effort derived copy; it is not evidence-backed durable ingestion.

Validated extraction journal

LLM ingestion now appends an EXTRACTION_VALIDATED operation before graph materialization. Its stable operation ID is derived from the source event ID; concurrent attempts retain the first committed result. Both backends verify the event’s owner, scope, and content hash before accepting a record. The operation contains structured grounded output, provider/model names, creation time, and explicit schema/grounding versions, excluding credentials and raw SDK responses. An indexing retry loads the saved output instead of calling the LLM again.

get_extraction(event_id, user_id=...) reads this record through the source owner boundary in the engine, sync client, HTTP and MCP. Empty extraction results are also recorded. A saved extraction does not mean graph materialization completed and lexical grounding does not establish semantic truth. Interrupted extraction jobs and saved derivation plans are recovered using RFC-0016. Explicit replanning can revise a derivation from the same saved extraction. Re-extraction under a new policy needs a future revision protocol; it must not overwrite this operation.

Current source-reading API

MemoryEngine.get_event(event_id, user_id=...) enforces optional owner scoping; MemoryClient exposes the same operation. get_event_nodes(event_id, user_id=...) requires an owner and reads graph evidence references directly, including retired nodes. This lookup avoids races from selecting the most recently created node. HTTP store and MCP store receipts use it to identify their own created node. HTTP event endpoints and MCP memory_get_event bind source access to the current principal. These APIs expose durable sources and current derivations; they do not claim complete graph replay from the event log.

Framework chat adapters represent clear, replacement and individual deletion as versioned system control events. They interpret those events when reading logical history, while prior message events remain immutable and owner scoped. Control events intentionally have no derived graph node or search-index entry; they are history operations rather than memories. Exact scope filtering prevents a clear in one scope from hiding messages in another. The logical history operation does not retire memory nodes already derived from prior message events; applications use the explicit owner-scoped lifecycle API when retrieval retirement is intended.

3. Event Log Schema

CREATE TABLE events (
    id             TEXT PRIMARY KEY,         -- UUID
    ts             TIMESTAMPTZ NOT NULL,
    stream         TEXT NOT NULL,
    actor_id       TEXT NOT NULL,
    actor_type     TEXT NOT NULL,            -- ActorType enum (RFC-0001 §5)
    role           TEXT NOT NULL,            -- Role enum (RFC-0001 §6)
    namespace_id   TEXT NOT NULL,
    content        TEXT NOT NULL,
    content_hash   TEXT NOT NULL,            -- SHA-256 hex
    metadata       JSONB,
    sequence_num   BIGINT NOT NULL,          -- Monotonically increasing within stream
    prev_hash      TEXT                      -- SHA-256 of prior event in stream. NULL for first event.
);
 
CREATE UNIQUE INDEX idx_events_stream_seq ON events(stream, sequence_num);
CREATE INDEX idx_events_ts ON events(ts);
CREATE INDEX idx_events_namespace ON events(namespace_id);
CREATE INDEX idx_events_actor ON events(actor_id);

prev_hash implements a tamper-evident chain within each stream. A verifier can confirm log integrity by walking the chain and re-hashing each record.

4. Operation Log Schema

Operations record actions taken on the memory system itself — object creation, transitions, scoring updates, and organiser decisions. They are distinct from Events (which record what happened in the world) but are stored in the same log.

CREATE TABLE operations (
    id             TEXT PRIMARY KEY,         -- UUID
    ts             TIMESTAMPTZ NOT NULL,
    op_type        TEXT NOT NULL,            -- See Section 6
    actor_id       TEXT NOT NULL,
    actor_type     TEXT NOT NULL,
    namespace_id   TEXT NOT NULL,
    target_id      TEXT,                     -- Object or event affected
    target_type    TEXT,                     -- 'memory_object', 'entity', 'edge', 'event'
    payload        JSONB NOT NULL,           -- Op-specific structured data
    policy_version TEXT NOT NULL,            -- Version of the policy that generated this op
    sequence_num   BIGINT NOT NULL,
    prev_hash      TEXT
);
 
CREATE INDEX idx_ops_target ON operations(target_id);
CREATE INDEX idx_ops_type ON operations(op_type);
CREATE INDEX idx_ops_ts ON operations(ts);

policy_version is mandatory. Every operation that results from a policy decision (decay, reinforcement, classification) MUST record which version of that policy was applied. This enables exact replay under the original policy when auditing.

5. Derived State Tables

Derived state is computed from events and operations. Implementations MUST treat derived state as a cache, not as authoritative. If any derived table is dropped and rebuilt by replaying the event + operation logs, the result MUST be identical to the prior state.

The following tables are derived:

-- Memory objects (derived from ASSERT and mutation operations)
CREATE TABLE memory_objects (
    id                TEXT PRIMARY KEY,
    type              TEXT NOT NULL,
    version           INTEGER NOT NULL,
    epistemic_type    TEXT NOT NULL,
    source_type       TEXT NOT NULL,
    evidence_ids      TEXT[] NOT NULL,
    asserted_by       TEXT NOT NULL,
    valid_from        TIMESTAMPTZ NOT NULL,
    valid_to          TIMESTAMPTZ,
    created_at        TIMESTAMPTZ NOT NULL,
    last_modified_at  TIMESTAMPTZ NOT NULL,
    lifecycle_state   TEXT NOT NULL,
    superseded_by     TEXT,
    confidence        REAL NOT NULL,
    salience          REAL NOT NULL,
    namespace_id      TEXT NOT NULL,
    value             TEXT NOT NULL,
    structured_value  JSONB
);
 
-- Graph edges (derived from RELATE operations)
CREATE TABLE edges (
    id          TEXT PRIMARY KEY,
    type        TEXT NOT NULL,
    source_id   TEXT NOT NULL,
    target_id   TEXT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL,
    created_by  TEXT NOT NULL,
    confidence  REAL NOT NULL,
    valid_from  TIMESTAMPTZ NOT NULL,
    valid_to    TIMESTAMPTZ
);
 
-- Entities (derived from ENTITY_CREATE and ENTITY_MERGE operations)
CREATE TABLE entities (
    id              TEXT PRIMARY KEY,
    type            TEXT NOT NULL,
    canonical_name  TEXT NOT NULL,
    aliases         TEXT[],
    namespace_id    TEXT NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL,
    last_seen_at    TIMESTAMPTZ NOT NULL
);

6. Operation Types

The following operation types MUST be supported. The payload field is operation-specific; its schema is defined per operation type.

Object lifecycle operations:

Epistemic operations:

Scoring operations:

Graph operations:

Organiser operations:

7. Integrity Verification

Implementations MUST support an integrity verification mode that:

1. Re-computes content_hash for each event and compares to stored value.
2. Walks the prev_hash chain for each stream and verifies the chain is unbroken.
3. Reports any discrepancies as integrity violations.

This verification MUST NOT be required for normal read operations (it is a background or on-demand check). Verification results MUST be logged as INTEGRITY_CHECK operations.

Implementations SHOULD support Merkle root computation over the event log for efficient remote verification.

8. Compaction Policy

The event log grows monotonically. Compaction is permitted under the following strict conditions:

When compaction is allowed:

• A set of events can be replaced by a logically equivalent summary operation without loss of information needed to reconstruct current derived state.
• The original events fall outside the configurable retention window AND are not referenced by any ACTIVE or SUPERSEDED memory object.
• A compaction operation is written to the log before the original events are removed.

When compaction is NOT allowed:

• Any event referenced by an ACTIVE memory object’s evidence_ids.
• Any event within the minimum retention window (default: 90 days).
• Any event in a stream where policy versioning has changed since the event was written (compacting across policy boundaries destroys auditability).

Compaction is a source of replay divergence if done incorrectly. Specifically: if the policy under which a DECAY_APPLIED or REINFORCE operation was calculated changes, and the underlying events are compacted away, the compacted state can no longer be verified to be correct under the new policy. The compaction log MUST record the policy version snapshot at the time of compaction.

9. Portability Artifact

The portability artifact is the exportable, self-contained representation of a memory state. It MUST be producible from any conforming implementation and loadable into any other.

Artifact structure:

memory_pack/
  manifest.json           -- Metadata, format version, namespace list, policy versions
  events.parquet          -- Full event log (columnar for efficiency)
  operations.parquet      -- Full operation log
  vectors/
    embeddings.bin        -- Binary embedding store
    index.hnsw            -- HNSW index
    embedding_meta.json   -- Model name, version, dimension
  graph/
    entities.parquet
    edges.parquet
  snapshot/
    memory_objects.parquet
    snapshot_ts.json      -- Timestamp of derived state snapshot
  checksum.sha256         -- SHA-256 of manifest.json + events.parquet + operations.parquet

manifest.json schema:

{
  "rms_version": "1.0",
  "format_version": "1",
  "created_at": "<ISO8601>",
  "namespaces": ["<namespace_id>", ...],
  "event_count": 0,
  "operation_count": 0,
  "policy_versions": {
    "decay": "<version>",
    "confidence": "<version>",
    "extraction": "<version>"
  },
  "embedding_model": "<model_name>",
  "embedding_version": "<version>",
  "is_full_export": true,
  "base_snapshot_id": null
}

Portability guarantee: Given the events.parquet and operations.parquet files and the policy versions recorded in the manifest, an implementation MUST be able to fully reconstruct all derived tables. The snapshot/ directory is a convenience cache only and is not authoritative.

10. Write Guarantees and Ordering

Events written in the same stream MUST be ordered by sequence_num. Implementations MUST guarantee:

• No two events in the same stream share a sequence_num.
• sequence_num is monotonically increasing within a stream.
• The prev_hash chain is consistent at all times.

For multi-writer scenarios (see RFC-0011), writes to the same stream are serialised at the storage layer. Implementations MAY use optimistic concurrency control with retry on conflict.

11. Cold Start Behaviour

A new memory system has an empty event log and no derived state. In this state:

• All retrieval queries MUST return empty results, not errors.
• The system MUST operate normally without preloaded data.
• Implementations MUST NOT pre-seed the event log with synthetic history.

This is the correct behaviour. An empty memory system that degrades gracefully to no-memory behaviour is preferable to one that fabricates history.

12. Conformance Requirements

[REQUIRED FOR TIER 1]

• Implementations MUST support all operation types in Section 6.
• The prev_hash chain MUST be maintained for every stream.
• policy_version MUST be present on every operation that results from a policy decision.
• Derived state MUST be rebuildable from events and operations alone.
• The portability artifact format in Section 9 MUST be producible and loadable.
• Compaction MUST NOT be performed on events within the minimum retention window.
• Compaction across policy version boundaries is NOT permitted.

13. Benchmark Requirement

Before this RFC progresses to Experimental status, implementers MUST publish:

• Write throughput benchmark: events per second at p95 for a representative workload.
• Rebuild benchmark: time to rebuild derived state from N events (N = 10k, 100k, 1M).
• Portability round-trip test: export artifact, load into a fresh instance, compare derived state SHA-256.
• Integrity verification speed: time to verify a log of N events.

End of RFC-0002

Source: docs/RFC-0002-Event-Store.md  in PRME at 58caddb1.