Your vector store is not a memory system

The failure

Here is a question from LoCoMo, the long-conversation memory benchmark — 10 multi-session conversations, 1,540 questions, 321 of them temporal:

We ran this through a pipeline most teams would recognize as good RAG: cosine retrieval over a wide candidate pool, Cohere Rerank v3.5 on top, GPT-4.1-mini answering. The #1-ranked memory it retrieved:

Retrieval worked perfectly. Out of ~14,000 memories, the single most relevant fact in the corpus was ranked first. The answer the model gave:

Note what kind of wrong: not a hallucination, not a retrieval miss. "Recently" is relative to a moment the chunk no longer carries. Embedding preserved what was said and discarded when, so the model anchored "recently" to the only timeline it had left — the day the question was asked.

This is the whole class of question that breaks flat retrieval: what did she say after the move? When did this happen? Which of these is still true? Anything whose answer depends on position in a timeline rather than similarity to the query.

Why similarity can't see time

Profiling retrieval on this benchmark, we logged the scores of the top 30 memories per query: every one ≈0.725, variance under 0.001. Conversational English embeds into a tight cluster; cosine distance can tell "about Gina's store" from "about Melanie's paintings," but within a topic, the geometry is flat. Nearest ≠ newest ≠ true-at-the-time-asked.

A reranker doesn't fix this, because relevance to a temporal question is a property of the timeline, and the timeline isn't present in any (query, chunk) pair the reranker scores.

The tuning ladder (we climbed all of it)

Before changing the architecture, we tried everything you'd try. A scoping note, because it matters: these rows are early single-pass runs — no reranker, a lighter answering path — so their absolute numbers are not comparable to the reranked configurations later in the post. The ladder measures one thing: whether prompt- and scoring-level fixes move the needle inside a fixed architecture. Temporal category, 321 questions, LLM-judge score 0–1:

Prompt grounding was a 13× improvement that still left the score at 0.04 on a 0–1 scale. The date-range boost — the "obvious" fix — went backward, and the audit explained why: 86% of temporal questions contain no parseable date at all. They're "When did X happen?", not "What happened on March 5th?" You cannot boost your way to an answer the scoring function cannot see. And a failure audit found 73% of temporal misses never reached the model's context at all — upstream of scoring, where re-ranking can't help.

That ~0.11 plateau is the ceiling of tuning, not of any pipeline's absolute score — a better answering path lifts flat and structured retrieval alike. The comparison that matters holds the pipeline fixed and changes only the structure.

Structure instead: the temporal tree

The fix is to stop storing memories as an unordered point cloud and start storing them as a tree where time is the topology. Every memory has a parent_id. At write time, a validator — the same LLM pass that already checks each incoming memory against its nearest neighbors — becomes the placement algorithm:

• Duplicate → rejected at the gate; never enters the tree
• Update / refinement → written as a child of the node it updates
• Contradiction → a fork: the old branch is marked superseded, the correction starts a sibling
• Unrelated → a new branch

Nothing enters the tree without a parent, so temporal questions become traversals instead of similarity lookups:

"What did she say after the move?" is a walk from the move-node toward the leaves. "Is this still true?" is a walk to the tip of a supersession chain. The date isn't metadata you filter on — it's the direction of growth.

The code

The read path for a subtree walk, as it runs in production (Memory.Fabric/Services/PostgresMemoryRepository.cs in our fabric layer):

WITH RECURSIVE tree(id, depth) AS (
    SELECT id, 0 FROM memories WHERE id = @rootId AND org_id = @orgId
    UNION ALL
    SELECT m.id, t.depth + 1 FROM memories m
    JOIN memory_signals ms ON ms.memory_id = m.id
    JOIN tree t ON m.parent_id = t.id
    WHERE t.depth < @maxDepth AND ms.is_stale = false
)
SELECT DISTINCT id::text AS "Id" FROM tree WHERE depth > 0

// Superseded rows are filtered at the OUTPUT, not inside the CTE: a superseded
// intermediate node must stay walkable or its whole live subtree vanishes with it.
return await _db.MemoryRecords
    .Where(m => ids.Contains(m.Id) && m.OrgId == orgId && !m.Signals.IsStale
        && (includeSuperseded || m.Signals.SupersededAt == null))
    .ToListAsync();

Deliberately boring — that's the point: once time lives in the structure, temporal logic is a recursive CTE, not a prompt. The subtle line is the comment: superseded facts stay walkable (history is what "before" questions are made of) but drop out of results unless asked for. A vector store can't express that distinction: a point is either in the index or it isn't.

What it costs

Placement isn't free: every candidate memory gets one validator LLM call before it enters the tree. Three things keep that affordable. It runs as a background job after the write returns, so write latency never sees it and burst ingest queues instead of blocking. Rejected duplicates cost a call but no storage. And the expense sits on the write side — reads, the side that scales with users, are the CTE above plus a vector lookup, no LLM in the loop. For this corpus: ~14,000 validation calls on a mini-tier model at ingest, zero at query time.

The number

The controlled comparison: same corpus, same Cohere v3.5 reranker, same answering model, same pinned judge; the flat row answers directly over its reranked pool, the tree row runs the full agent path:

Per-question, on the temporal split: 85 questions flip from wrong to right when the tree replaces flat retrieval; 18 flip the other way; net +67 of 321. Gina's ad campaign is one of the 85 — the tree run answers "around January 29, 2023."

The attribution is worth staring at: from flat cosine with no reranker (0.374 overall), Cohere adds +0.18; the tree adds another +0.22 on top. The structure contributes more than the best commercial reranker on the market — and they stack, because they fix different failures.

The 18 losses are worth as much as the 85 wins, so we read all of them. Seven are coverage losses — the branch walk never surfaced a fact the flat scan found; that's the failure mode structure adds: pick the wrong branch and the fact is invisible. Five are pipeline artifacts (the agent leaked reasoning into its answer; the judge scored the verbiage). Three are judge noise — in two, the tree gave the same date as the flat run and was scored wrong; in one, the flat run's wrong year was scored right. Three picked an adjacent event on the correct branch. The one real regression is branch selection, and it costs a fraction of what the structure buys.

What a memory system actually is

A vector store is a similarity index. Necessary — it's how you find the right neighborhood fast — but an index is not a memory. Memory is ordered, self-revising, and aware of what superseded what. Those properties have to live in the storage structure, because no prompt can reconstruct an ordering the embedding already threw away.

That's the design rule we build Aivery around: retrieval finds the neighborhood, structure does the remembering. Everything a raw vector store gets confidently wrong about time, a temporal tree gets right by construction.