How the knowledge pipeline works

The knowledge pipeline is the path from "a session just happened" to "six months from now I can ask a semantic question about it and get the right answer". It is the long-term arm of the memory model.

Four moving parts feed it:

• session-export — writes clean markdown sessions into the store's sessions/.
• compound-loop — writes distilled learnings into the store's learnings/.
• QMD (optional external tool) — indexes the knowledge tree semantically and exposes a qmd query interface.
• The knowledge store itself — a plain filesystem tree, and a git repository, that the user can backup, grep, and open in any editor.

See ADR-011, ADR-016 for the design decisions. This page explains the mechanics.

Store shape

<store root>/                   # $LAZY_KNOWLEDGE_ROOT, [knowledge].root, or the default
├── knowledge.toml              # the marker: declares the three area names
├── sessions/                   # written by session-export hook
│   ├── 2026-03/
│   │   ├── 2026-03-01-a1b2c3d4.md
│   │   ├── 2026-03-02-f5e6d7c8.md
│   │   └── ...
│   └── 2026-04/
│       └── 2026-04-13-9a8b7c6d.md
├── learnings/                  # written by compound-loop worker
│   ├── 2026-03/
│   │   └── 2026-03-15-file-based-queue-is-enough.md
│   └── 2026-04/
│       └── 2026-04-13-symlinks-vs-copies-for-profile-deploy-laptop.md
└── memory/                     # per-project distilled memory, keyed by git remote
    └── github.com/
        └── <owner>/
            └── <repo>/
                ├── MEMORY.md
                ├── decisions.jsonl
                ├── failures.jsonl
                └── ...

The marker is the only declaration of the layout

[knowledge]
version   = 1
sessions  = "sessions"
learnings = "learnings"
memory    = "memory"

sessions and learnings are required — a missing one raises. memory is optional and defaults to "memory": it was added after the first stores existed, and bumping version instead would have broken every one of them at once over a directory they did not use yet. lh knowledge init writes all three.

Configuration resolves exactly one thing: where the store root is. How it is laid out inside comes from knowledge.toml, and from nowhere else. The split is deliberate — a root is environmental and differs per machine, while a layout is global and must be identical everywhere. Two independent declarations of the same layout is precisely how a directory named learnings ends up configured as Learnings, which works on a case-insensitive filesystem and breaks on Linux.

Validation is loud on every failure: a missing file, an unknown version, an absolute or escaping subdirectory name, or a missing field all raise rather than fall back. A field read as "" would silently land files at the repository root.

Every file is plain markdown with YAML frontmatter. No database, no lock files, no hidden index. You can rsync -a the whole directory to backup, and rg it directly if you do not have QMD.

Session export in detail

Module: src/lazy_harness/knowledge/session_export.py. Called from the session-export built-in hook on every Stop event.

What gets parsed from the JSONL

Claude Code stores each session as a JSON-per-line file:

{"type":"permission-mode", ...}         # interactive marker (or "last-prompt"), anywhere in the file
{"type":"system","cwd":"...", ...}      # system message with metadata
{"type":"user","message":{"content":"..."}, ...}
{"type":"assistant","message":{"content":[{"type":"text","text":"..."}]}, ...}
...

_parse_session_jsonl walks this and produces:

• meta: {cwd, version, branch, timestamp} from the first system record.
• messages: a list of {role, text, timestamp} dicts. Content extraction handles both string content ("hello") and structured content ([{"type":"text","text":"..."}]).
• is_interactive: set when a permission-mode or last-prompt record appears anywhere in the file, or, for a Codex transcript, a user response_item. Without it the export is skipped as non-interactive (unless forced). The marker is not exclusive to interactive runs: Claude Code writes last-prompt in headless claude -p runs too, so this gate does not by itself keep them out; only the min_messages threshold below does. The compound loop's own evaluator sessions are therefore rejected earlier, before any other gate and even when forced: export_session returns evaluator when is_evaluator_session finds the evaluator prompt as the first user message.

The interactive + length filter

A session is only exported if it has at least min_messages (default 4) messages after the interactive filter. Below that threshold, the export function returns None and the hook logs a skip.

Classification rules

_classify(cwd, rules) returns (profile, session_type) based on the first rule whose pattern is a case-insensitive substring of cwd. Rules come from [[knowledge.classify_rules]] in config.toml; if the section is omitted, a built-in default list applies. See ADR-028.

Default rules (used when nothing is declared):

[[knowledge.classify_rules]]
pattern = "lazymind"
profile = "personal"
session_type = "vault"

[[knowledge.classify_rules]]
pattern = "obsidian"
profile = "personal"
session_type = "vault"

[[knowledge.classify_rules]]
pattern = "/repos/lazy/"
profile = "personal"
session_type = "personal"

[[knowledge.classify_rules]]
pattern = "/repos/flex/"
profile = "work"
session_type = "work"

Override entirely by declaring your own block — for example, a consultant juggling client work:

[[knowledge.classify_rules]]
pattern = "/srv/clients/acme/"
profile = "client"
session_type = "acme"

[[knowledge.classify_rules]]
pattern = "/opt/research/"
profile = "research"
session_type = "experiment"

Order matters: the first matching pattern wins. Anything that matches no rule is classified as ("other", "other"). Setting classify_rules = [] opts out completely. The profile value also drives the profile: frontmatter field, so context-inject and other downstream filters can discriminate without scanning bodies.

Project name extraction

_extract_project walks from cwd upward looking for the first .git directory and returns that directory's basename. If no git root is found, it falls back to basename(cwd). This is how a session in /Users/me/repos/lazy-harness/src/lazy_harness/hooks/ correctly identifies lazy-harness as the project.

There is a separate decoder, _decode_project_dir, that reverses Claude Code's directory-name mangling (/ replaced with -). Because hyphens appear inside real directory names (lazy-harness, lazy-claudecode), naive replace("-", "/") is wrong — the decoder tries candidate splits against the real filesystem and picks the one that exists.

The written file

Output path: <store>/sessions/YYYY-MM/YYYY-MM-DD-<short_id>.md (short_id = first 8 chars of session_id).

Frontmatter:

---
type: claude-session
session_id: 9a8b7c6d-1234-5678-90ab-cdef12345678
date: 2026-04-13 18:32
cwd: /Users/me/code/lazy-harness
project: lazy-harness
profile: personal
session_type: personal
branch: docs/restructure
claude_version: 1.2.3
messages: 47
---

Body: a sequence of ## User and ## Claude blocks with the extracted text, separated by blank lines. Tool-use blocks are intentionally not included — they would explode file size and are already captured by pre-compact for the sessions where file-touch history matters.

Idempotence

_existing_message_count reads the frontmatter of any pre-existing export at the same path. If the on-disk copy already contains at least as many messages as the current JSONL, the export is skipped. This makes the hook safe to re-run — for instance after a resume, or during a lh migrate replay.

Atomic writes

Every write goes through _atomic_write: write to <dir>/.filename.tmp, then os.replace to the target. A sync observer (iCloud, Dropbox) sees a single rename event rather than a window during which the file is partially written. Required whenever the store lives under a synced folder.

Learnings via compound loop

The compound-loop worker writes one markdown file per learning into the store's learnings subdirectory. The structure is identical in spirit to session exports: year-month subdirectories, dated filenames, YAML frontmatter. Learning filenames carry a trailing -<host> slug so that two machines distilling the same title on the same day produce two files rather than one conflict.

For the full flow, see how the memory compound loop works. The key points for the pipeline:

• Learnings are write-once. Existing files are not overwritten. This is intentional — a learning is a snapshot of a specific session's insight, and editing it later changes the historical record.
• Deduplication happens in the LLM prompt, not in the filesystem. The worker passes the titles of the last 50 learnings into the prompt with explicit "do not generate semantic duplicates" instructions.
• Learnings carry a scope field (universal | backend | infra | consulting) that lets future queries slice by applicability.
• Nothing prunes or merges near-duplicates after the fact. The tree is append-only by construction and reviewing it is a manual read — the framework ships no learnings-review job.

Transport: the store is a git repository

The store is checked out on every machine that writes to it, and a scheduler job runs lh knowledge push on each of them. One cycle is:

1. Take an exclusive flock. A held lock means another cycle is running, so this one exits 0 silently.
2. Validate knowledge.toml. An unusable marker stops here, before git is touched at all.
3. Make sure .gitattributes carries merge=union for *.jsonl and claude-md.proposal.md, adding only the missing lines. A store with no remote skips this step.
4. add -A, then commit with a subject counting what changed and naming the host. Nothing to commit and nothing unpushed is not an early exit: the cycle still fetches.
5. fetch, then merge --no-edit of the upstream. A clone that is only behind fast-forwards and reports updated; a diverged one gets a single merge commit, so a long divergence is never replayed commit by commit.
6. push. On failure the commits stay local and the next cycle retries.

No producer ever calls this. The session-export hook and the compound-loop worker only write files, which is what makes the transport failure-tolerant: with no network, no credentials, or a dead remote, the writes still land on disk and the next cycle picks them up. No session ever blocks on a network syscall.

Most writes create a new file under a unique key, and the host suffix on learning filenames keeps those keys unique, so two clones union without conflict. Three kinds of file under memory/<host>/<org>/<repo>/ are modified instead, and each has a rule:

File	Written by	Merge rule
*.jsonl (decisions, failures, grades)	appended	union: both sides' lines are kept
claude-md.proposal.md	appended	union: both sides' lines are kept
handoff.md	overwritten whole	newest wins, by written_at

For handoff.md the cycle reads the timestamp from either format found in the wild, the written_at: frontmatter key or the - Written at: body line, and compares them as timezone-aware datetimes, so differing offsets are handled. A conflict on any other path, or a handoff whose timestamp is missing, unparseable or without an offset on either side, aborts the whole merge (merge --abort) and exits non-zero; nothing is resolved blindly.

Failure	Consequence	Data lost
No network at push	local commits accumulate	no
No network at fetch	commits stay local, next cycle retries	no
Conflict outside handoff.md, or a handoff with no usable timestamp	merge aborted, paths logged, stays pending	no
Lock held	cycle skipped	no
Invalid marker	loud failure, nothing written	no

QMD integration

QMD is an external semantic search tool the framework integrates with but does not require. Module: src/lazy_harness/knowledge/qmd.py.

How the framework calls QMD

One entry point: run_qmd(action, collection, timeout) — a thin wrapper around subprocess.run(["qmd", action, "--collection", collection]). Three convenience functions:

• sync(collection) — calls qmd update, which re-parses markdown files and updates indices.
• embed(collection) — calls qmd embed, which computes embeddings for newly synced files.
• status() — calls qmd status, which reports index health.

The session-export hook triggers qmd update automatically at the end of every exported session. Embedding is left to scheduled jobs — it is more expensive and does not need to happen synchronously.

QMD availability and graceful degradation

Every call site starts with shutil.which("qmd"). If QMD is not installed, the framework silently skips the indexing step. All other parts of the pipeline still work: sessions are still exported to markdown, compound loop still writes learnings, context-inject still reads files directly. What you lose is cross-session semantic recall — qmd query "circular imports python" is the feature that is unavailable.

QMD can be installed later without re-running any framework command. The next session-export invocation will pick up the binary and start triggering qmd update.

Collection configuration

QMD treats each indexed directory as a named collection. The framework's convention is that each store subdirectory is its own collection — raw transcripts and distilled learnings are corpora of very different value, and a single collection makes the distilled one unreachable behind the raw one — and the collection name is configured externally in QMD itself (not in lazy-harness). This keeps the framework unopinionated about QMD's config model — we call qmd the same way you would call it by hand.

Querying the knowledge tree

With QMD:

qmd query "when did I last deal with symlinks on macOS"
# returns matching session/learning files with scores

Without QMD:

rg -l "symlink" ~/Documents/lazy-harness-knowledge/
# or
find ~/Documents/lazy-harness-knowledge -name '*.md' | xargs rg 'symlink'

Both work. One is faster and semantic; the other is always available and a good fallback.

Inside a session, Claude Code itself can also read these files as regular markdown — which means an agent with filesystem access can answer "what did we learn about X last quarter" by reading the learnings directory directly, without needing QMD as a tool.

Backup and portability

Because the entire knowledge pipeline outputs to a single directory of markdown files:

# Backup
rsync -a ~/Documents/lazy-harness-knowledge /backup/location/

# Restore
rsync -a /backup/location/lazy-harness-knowledge ~/Documents/

# Git-version it
cd ~/Documents/lazy-harness-knowledge && git init

Moving to a new machine is "point [knowledge].root in config.toml (or $LAZY_KNOWLEDGE_ROOT) at the clone and run qmd update". Nothing inside the framework holds state that can diverge from the directory contents — the layout comes from the store's own knowledge.toml, and each project's memory is keyed by its git remote rather than by where the checkout sits on this machine.

Other consumers of the same pipeline

QMD is the headline consumer of the knowledge tree, but it is not the only one. Two more pieces plug into the same on-disk artifacts and are worth understanding alongside the main pipeline.

Engram persistence loop — JSONL mirrored into a per-project store

Module: src/lazy_harness/knowledge/engram_persist.py. Wired as a Stop hook (engram-persist) that runs after compound-loop writes its new entries. Design rationale: ADR-029.

While the JSONL pair (decisions.jsonl / failures.jsonl) is the file-of-record, Engram is the agent's MCP-queryable view of the same data. The mirror keeps both views in sync without dual-writing from compound-loop itself — compound-loop only writes to JSONL, and engram-persist reads what is new and ships it to engram save.

Determinism comes from a per-file byte cursor, one per project per machine. With memory in the knowledge store it lives in <lh data dir>/engram-cursors/<memory dir relative to the store>/engram_cursor.json, shared by every profile and never synced; memory outside the store keeps a per-profile cursor under <agent dir>/engram-cursors/:

{
  "version": 1,
  "decisions_offset": 18234,
  "failures_offset": 5120,
  "updated_at": "2026-04-13T18:33:01Z"
}

Each run, for each kind:

1. Open the JSONL, seek to the stored offset.
2. Read whole lines until EOF. Partial lines are deferred (the writer might still be flushing the entry).
3. Decode each line as JSON; malformed lines advance the cursor and are counted as skipped_malformed — they are not retried.
4. Call engram save --type <kind> --project <project_key> --scope project -- <title> <json> — flags first, so a summary starting with -- is not read as a flag. The project key is the parent of git rev-parse --path-format=absolute --git-common-dir, taken by name — so a linked worktree collapses onto the repository that owns it rather than registering as its own project.
5. On success, advance the cursor (atomic tempfile + os.replace). At most 25 saves are attempted per run (MAX_SAVES_PER_RUN); the rest drains on later runs.
6. On failure, leave the cursor untouched and stop processing this kind. The next run picks up from the same offset → at-least-once delivery.
7. After five failed exits for one entry (MAX_ATTEMPTS_PER_ENTRY; timeouts do not count), move it to engram_quarantine.jsonl beside the cursor and advance, so it cannot block the entries after it. Ordering holds for every entry but a quarantined one.

This produces four things on disk besides the Engram DB itself:

• engram_cursor.json — offsets per kind and failed attempts at a stuck entry, atomically updated, in the location above.
• engram_quarantine.jsonl — beside the cursor, the entries that ran out of attempts, with their raw line and last error.
• <agent runtime dir>/logs/engram_persist.log — append-only error log, one line per subprocess failure (both output streams), quarantine or missing binary.
• <agent runtime dir>/logs/engram_persist_metrics.jsonl — one record per run (run summary) plus one per slow engram save (≥ 500 ms). Run records include whether the save cap was reached and whether the cursor file advanced. lh doctor reads these with age, failure rate, cursor lag, and quarantined entries; a capped run with persisted progress and no failed save reports catching up instead of failing for large lag.

If engram is not on PATH, the loop logs a no-op and exits cleanly. Like every other external integration in the framework, this layer is opt-in: install engram, declare [memory.engram].enabled = true, the rest of the pipeline keeps working unchanged.

Structural layer — Graphify

Module: src/lazy_harness/knowledge/graphify.py. Pinned to a specific Graphify version (see ADR-023) so that an upgrade is an explicit decision rather than a transparent behaviour change.

Graphify is a tree-sitter–based code-structure indexer covering 25 languages. Where QMD answers "where did we discuss X across all my notes?", Graphify answers "what calls this function?", "explain this module", or "show me the dependency neighbourhood of this symbol". Its output is a JSON graph plus an interactive HTML report under graphify-out/ inside each repo, not in the knowledge store.

The harness's job is integration, not re-implementation:

• is_graphify_available() probes shutil.which("graphify"). If absent, every other piece of the framework keeps working — the structural layer is simply missing.
• check_version() parses graphify --version and compares against PINNED_VERSION. lh doctor surfaces a row when the installed version drifts from the pin.
• run_graphify(action, target, timeout) is a thin wrapper around subprocess.run(["graphify", action, target]), used by the /graphify skill in interactive sessions. The framework never invokes Graphify automatically — building the graph is up to the user (or a scheduled job, or a post-commit hook).

Configuration lives under [knowledge.structure] in config.toml:

[knowledge.structure]
engine = "graphify"
enabled = true
version = "0.9.67"

enabled = true is what lh deploy reads to wire the Graphify MCP entry into each profile's agent config, provided the graphify-mcp binary is present. Rebuilds are demand-driven: run graphify update <path> to refresh a graph from the AST without an LLM call, or graphify check-update <path> to test whether one is due.

Because graphify-out/ is checked into the repo by convention, teammates and future sessions reuse the index without rebuilding. Adding it to .gitignore is a deliberate (if rare) choice for repos where build cost dominates over reuse.