How the metrics ingest pipeline works¶

lh status shows sessions, tokens, and cost by reading a SQLite database at ~/.local/share/lazy-harness/metrics.db — data_dir(), which honours LH_DATA_DIR then XDG_DATA_HOME — or wherever [monitoring].db points when that is set. That database does not populate itself — something has to parse the agent's session JSONLs and feed it. That something is the metrics ingest pipeline: a standalone module (lazy_harness.monitoring.ingest) exposed as lh metrics ingest, designed to produce numbers that reconcile with npx ccusage without ever double-counting tokens.

This page explains what the pipeline does, how it guarantees precision, and how to wire it into the scheduler so lh status stays live.

Producer and sink¶

Two pieces are needed on either side of the pipeline:

• Producer — the profile's own agent, through the TranscriptReader Protocol. agent.read(path) yields one TranscriptEvent per signal occurrence, and ingest keeps the TOKEN_USAGE ones: each carries a TokenUsage (the four token buckets, plus the 1-hour cache write where the provider splits it), the model that produced the turn, and the provider's own message_id for it. There is no dialect-specific parser here any more — Claude Code's message.id and Codex's response_id arrive through the same field, and an agent that implements the Protocol is metered whatever it is called. A companion Protocol, TranscriptIdentity, answers which session and project a transcript belongs to; a reader that does not implement it falls back to the file's stem and no project.
• Sink — lazy_harness.monitoring.db.MetricsDB owns the SQLite file. session_stats is keyed by UNIQUE(session, model) and stores per-bucket token counts plus a pre-computed cost. upsert_stats(entries) writes each row with INSERT … ON CONFLICT(session, model) DO UPDATE, so re-ingesting a session overwrites its totals in place while sessions absent from this run are left untouched.

The walk¶

ingest_all(cfg, db, pricing) iterates every configured profile via list_profiles(cfg). For each profile it calls ingest_profile(profile, db, pricing) which:

1. Resolves the transcript directory the profile's own agent declares — <config_dir>/<agent.session_dirs()["sessions"]>, which is projects/ for Claude Code and sessions/ for Codex, and is unrelated to the metrics database's location. Two cases are skipped rather than walked: a profile whose directory doesn't exist or whose agent declares no sessions directory at all, and one whose agent implements no TranscriptReader. The second is a capability test, not a name — an agent nobody has written a reader for is skipped, and lh doctor's Transcripts section carries the verdict, because walking it would find nothing and report the profile empty, which is indistinguishable from a profile that did no work.
2. Asks the agent for its transcripts (locate_sessions), which for Claude Code means every *.jsonl under the directory recursively, including nested subagent files at <session-uuid>/subagents/agent-*.jsonl, and for Codex means sessions/YYYY/MM/DD/rollout-*.jsonl. Paths that sit under a memory/ ancestor are excluded here rather than in the reader — those are this harness's own episodic logs (decisions.jsonl, failures.jsonl), so what to do with them is the consumer's business and not the agent's.
3. Sorts the collected files by st_mtime_ns ascending. locate_sessions promises no order, and dedup needs one: older files attribute their messages first, so the canonical ownership is stable across runs.
4. Iterates the files in order, maintaining a seen_msg_ids: set[str] across the whole profile. Each event's message_id is checked against the set; novel events bump an in-memory aggregator keyed by (session_id, model); already-seen ones are counted as deduped and dropped. An event whose provider gives no stable id is counted every time it is seen — there is nothing to match on, and losing real tokens is worse than double-counting a resume's shared prefix.
5. Each response is offered independently to API-equivalent pricing before its money is aggregated; this preserves context-class and effective-date differences. The context class is derived per response from the usage the record reports: OpenAI's 272K boundary (ADR-067) and Haiku 5.5's 100K boundary (anthropic_context_class, ADR-070). Billed pricing sums tokens per (session, model, context_class) bucket and prices each bucket with cost_for_billing_model() over DEFAULT_PRICING plus any [monitoring.pricing] override. The buckets are summed back, so the stored rows stay per (session, model) and are handed to upsert_stats(entries). Sessions whose transcripts no longer exist on disk are not re-scanned, so their rows remain beyond transcript retention.

The whole pass is summarized as an IngestReport with the following counters: sessions_scanned, sessions_updated, sessions_skipped, messages_total, messages_deduped, and any per-file errors. lh metrics ingest prints the headline counters as the last line of output.

Ingest is fail-soft per file. A file that raises while being dated, identified, read or parsed — a foreign JSONL dropped under a profile's sessions tree by something other than the agent, a half-written transcript, a permissions error — is recorded as "<path>: <exc>" in errors and the walk moves on to the next file; it never aborts the profile or the run. The headline line always shows an errors N count when N > 0 so a fail-soft run is never mistaken for a clean one; the individual paths only print under --verbose.

flowchart LR
  A[lh metrics ingest] --> B[load config.toml]
  B --> C[open MetricsDB]
  C --> D{for each profile}
  D --> D2{agent is a\nTranscriptReader?}
  D2 -- no --> D3[skip - doctor reports it]
  D2 -- yes --> E[agent.locate_sessions]
  E --> F[skip memory/*]
  F --> G[sort by mtime asc]
  G --> H[agent.read - TOKEN_USAGE events]
  H --> I{message_id in seen?}
  I -- yes --> J[drop - deduped]
  I -- no --> K[aggregate by session, model]
  K --> L[price each response]
  L --> L2[aggregate tokens and priced money]
  L2 --> M[upsert_stats per session,model]
  J --> H
  M --> D

Why it can't double-count¶

Three independent guarantees stack up:

1. Message-id dedup across files¶

When Claude Code /resumes a conversation, it writes a new JSONL whose first section re-includes every prior message. Without dedup, the shared prefix gets counted once per resume chain — for a conversation resumed four times, that's 5× overcounting.

The pipeline defends against that with seen_msg_ids: each TranscriptEvent.message_id — Claude Code's upstream message.id, Codex's response_id — is attributed to exactly one (session_id, model) bucket, the first one the walk sees it in, which is the oldest file by mtime. Every subsequent occurrence in a resumed JSONL is skipped and counted under messages_deduped.

In production this matters a lot: measured on a real install, ~50% of assistant messages in ~/.claude-* projects are duplicates introduced by resumes. Dedup is the difference between matching ccusage and being off by ~3×.

2. Append-only source of truth¶

Claude Code only ever appends to an existing session JSONL, never rewrites past messages. So re-parsing the full file always returns the exact cumulative totals as of now — not a delta, not a snapshot — and re-attributing them via dedup is idempotent by construction.

3. In-place upsert, never accumulate¶

upsert_stats() writes each (session, model) row with ON CONFLICT … DO UPDATE, replacing the stored totals with the freshly-recomputed ones. Re-ingesting the same session never adds to a running sum — it overwrites. Combined with (2), parsing a grown transcript yields the new cumulative total, not old+new. Sessions missing from a run keep their last-written row, so a transcript pruned by retention does not erase its cost from lh status.

Pricing¶

DEFAULT_PRICING in monitoring/pricing.py holds per-million-token rates for the Claude models currently observed in the wild, taken from Anthropic's published pricing table. Mirror that table, not a third-party mirror of it: the claude-sonnet-5 row was once copied from claude-sonnet-4-6 and over-reported every Sonnet 5 session by 50% until the two were compared against the source.

Cache rates follow the published multipliers, relative to the model's base input rate:

Cache writes are priced by TTL¶

Claude Code reports which TTL a cache write used, under usage.cache_creation:

"cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 26038}

split_cache_creation() in monitoring/collector.py is the single place that reads it, and both transcript readers go through it. The two buckets stay separate as far as calculate_cost(), which prices each at its own rate.

They are recombined immediately after. The stored session_stats.cache_create column and the emitted MetricEvent carry one total cache-write token count, as they always have — only the cost changes. Splitting the column would cost a schema migration and a wire-format bump for a distinction no reader consumes; if re-pricing history ever needs it, the split can be added then.

A transcript written before the breakdown existed carries only cache_creation_input_tokens. Nothing records its TTL, so it is billed as a 5-minute write — the alternative invents a 2× charge on evidence we do not have.

This distinction is not marginal. Across a week of measured local traffic, 1-hour writes were 92.9% of all cache-write tokens; pricing them at the 5-minute rate under-reported the total cost by 26.5%.

Haiku 5.5 is priced by prompt length¶

A request whose prompt (input + cache_read + cache_create + cache_create_1h) is over 100,000 tokens pays the higher tier on every bucket, output included. Every other current Claude model is priced flat across its window.

LONG_CONTEXT_PRICING holds the tier and anthropic_context_class classifies each response. Ingest prices each class bucket before summing, so stored rows stay per (session, model). For billed cost, a [monitoring.pricing] override of the model's row disables the shipped long tier for it; API-equivalent cost always prices from the published list (DEFAULT_PRICING and LONG_CONTEXT_PRICING), so it keeps the long tier.

Launch discounts expire on their own¶

A model that ships with an introductory rate gets an entry in INTRODUCTORY_PRICING, carrying the discounted rates and the window they apply to:

INTRODUCTORY_PRICING = {
    "some-future-model": IntroductoryRate(
        since="2027-01-01",
        through="2027-03-31",
        rates={"input": 2.0, "output": 10.0, "cache_read": 0.2, "cache_create": 2.5},
    ),
}

DEFAULT_PRICING always holds the standing rate, so the discount is the exception and reverting is the default. calculate_cost() takes the session's date and picks the discounted rate only for sessions inside the window — which means a session from the discount period keeps its discounted cost forever, even when re-ingested years later, and a session after it never gets one.

Encoding the end date is what makes this automatic. A comment saying "bump this after the window closes" is not a mechanism: nothing reads it, and the day it lapses every cost figure silently drifts.

The table is empty today. Sonnet 5's launch discount became its standard price — Anthropic cancelled the increase that had been scheduled for 2026-09-01 — so the rates moved into DEFAULT_PRICING and the window was removed. An expiry that still fired would have inflated every reported Sonnet 5 cost by 50% overnight, with no change in usage. A window is only correct while the vendor still intends to raise the price; re-check the source before trusting one.

Unpriced models¶

calculate_cost() returns 0.0 for any model absent from the table. Since a genuinely free session also costs 0.0, ingest_profile() collects every such model into IngestReport.unknown_models, and lh metrics ingest prints them.

Placeholders are exempt. Claude Code writes <synthetic> in the model field of messages that consumed no tokens; is_pseudo_model() treats any angle-bracketed name as one of these, and they never reach the warning. Without that carve-out the warning fires on every ingest of every real session tree — which is how a warning stops being read.

What you do see is a genuine gap:

! no pricing for claude-future-model-99 — those sessions were priced at $0.
  Add rates under [monitoring.pricing] in config.toml.

This is not gated behind --verbose: an unpriced model means billed-cost coverage is incomplete, and that should be impossible to miss.

Overrides¶

You can override any model's rates per-install under [monitoring.pricing] in config.toml:

[monitoring.pricing."claude-opus-4-6"]
input = 5.0
output = 25.0
cache_read = 0.5
cache_create = 6.25
cache_create_1h = 10.0

Overrides are merged over DEFAULT_PRICING at ingest time via load_pricing(), and they win over an introductory window — once you set a model's rate yourself, that rate is what gets used on every date.

An override replaces the model's whole rate dict. One written before cache_create_1h existed carries no 1-hour rate, so calculate_cost() falls back to 2× the override's input rather than billing that bucket at zero.

Inference calls¶

Not every billed call leaves a transcript. The compound-loop evaluator runs claude -p --no-session-persistence, so there is nothing for this walk to find. For those calls run_inference is the producer: a backend that reports usage (today the claude backend, from its --output-format json result envelope) hands it to llm/usage.py, which builds the same MetricEvent this pipeline would build from a transcript of that one response, priced by the same functions, and writes it through the same sinks.

• One row per model the call billed, keyed by the call's own session_id.
• workload is inference:<role> (the evaluator is inference:distill) and agent is the backend type, so lh status tokens --by workload separates evaluator spend from user sessions.
• profile and billing_model come from the profile the call runs under; project is empty.
• Spend, not a session: every reader that counts sessions skips these rows (one predicate, monitoring/inference_rows.py), while tokens and cost still sum them.
• Recorded only when [monitoring].enabled is true, and fail-soft: a metrics failure never changes the inference result. A remote sink that cannot be built (an http_remote with neither url nor url_env) does not take the local row with it: each sink is built on its own, sqlite_local still gets the row, and one line naming the failed sink goes to stderr.
• Remote sinks get the row in their outbox; it is delivered on the next lh metrics ingest drain.

A backend that does not report usage (the OpenAI-compatible ones today) records nothing.

Precision tests¶

The pipeline is covered by tests/unit/test_ingest.py. The invariants worth calling out:

• test_ingest_profile_upserts_totals — happy path: one session, one ingest, one row with the right totals.
• test_ingest_is_idempotent — running ingest twice yields identical stored totals, no doubling.
• test_ingest_reflects_session_growth — append a new assistant turn to the JSONL, re-ingest. The stored row reflects the new total, not old+new.
• test_ingest_preserves_sessions_after_transcript_pruned — a session's row survives once its transcript is deleted by retention, so cost history does not shrink to a rolling window.
• test_ingest_dedups_messages_shared_across_resumed_sessions — two files share a message.id; it is counted exactly once.
• test_ingest_discovers_subagent_files — nested <uuid>/subagents/agent-*.jsonl files are found and counted.
• test_ingest_skips_memory_jsonls — memory/decisions.jsonl and memory/failures.jsonl are ignored even if they happen to contain assistant-shaped lines.
• test_ingest_isolates_profiles — two profiles with different sessions don't contaminate each other.

If any of these break, the numbers in lh status stop being trustworthy. They run on every uv run pytest.

Running it on a schedule¶

The harness already has a scheduler abstraction (lh scheduler) that bridges [scheduler.jobs.*] entries in config.toml to launchd plists on macOS. The metrics ingest is a normal job entry — no special support needed. A typical setup:

[scheduler.jobs.metrics-ingest]
schedule = "*/15 * * * *"
command = "/Users/you/.local/bin/lh metrics ingest"

Then lh scheduler install to register the job with the platform backend and lh scheduler status to confirm it is loaded. This works on every supported platform. On a headless Linux machine, check lh selftest afterwards: user timers do not fire once your last session ends unless lingering is enabled. Because the pipeline re-derives each session's totals from its transcript and the file reads are cheap, a shorter cadence is only bounded by how fresh you want lh status to be.

Pair this with whatever manual ingest you want: running lh metrics ingest at the end of a noisy day gives the same final state as letting the cron tick through the day on its own. The pipeline is deterministic.

When you should not use a hook¶

A tempting alternative is to ingest from the Stop / session-export hook so the DB updates instantly when a session closes. The reason the harness ships with a CLI + cron instead is decoupling: a SQLite write failure inside a hook would leak into the agent's perceived session cleanup, and the hook runs even when monitoring is disabled. Keeping ingest out-of-band means a broken ingest never breaks a session.

If you want live updates anyway, nothing stops you from calling lh metrics ingest from a local post-stop hook — just accept that you own the failure mode.

The sink layer¶

Ingest writes session-level rollups (session_stats) to one SQLite file. The sink layer is the orthogonal channel for per-event telemetry — one record per assistant message, fan-out to one or more destinations. This is what the framework uses to ship metrics to a remote backend, with on-disk buffering and retry, without coupling the ingest pipeline to any of it.

Built-in sinks¶

Two sinks ship with the framework:

The list of active sinks is declared in [metrics].sinks. With no [metrics] block, the default is ["sqlite_local"] and the framework does zero network I/O — that property is load-bearing for offline-friendly setups, so adding a new built-in sink that talks to the network without an explicit opt-in is rejected by the config parser.

Why an outbox¶

http_remote.write() does not POST. It only enqueues. Three reasons:

1. Hooks must be fast. A Stop hook that blocks on a 5-second HTTP timeout drags out every session close. Enqueue is microseconds.
2. Network is unreliable. A failed POST should not lose the event. Persisting to SQLite first means the worst case is "send later", not "send never".
3. Multiple lh processes. Two terminals running lh in parallel both ingest, both enqueue. Without an outbox, both would also send — duplicating events on the backend.

The outbox solves (3) with a claim-with-lease protocol: when a worker drains a batch, it stamps a 60-second lease on those rows. Other lh processes skip leased rows. If a worker crashes mid-send, the lease expires and the next drainer picks them up.

Drain — when and how¶

Two triggers fire drain_http_remote:

1. Opportunistic. Every normal lh metrics ingest drains after it ingests, in the same process. This means the cron-driven ingest cadence is also the drain cadence, with no extra wiring. --dry-run parses into an in-memory database without writing to the configured database, enqueuing events, or contacting remote sinks.
2. Explicit. lh metrics drain runs only the drain phase. Useful for catching up after a backend outage without re-running the full ingest.

The drainer iterates the claimed batch, POSTs each event, and writes back per-row state:

A sink that cannot be built is handled the same way: lh metrics ingest warns on stderr, naming the sink, and ingests into the sinks that did build instead of aborting. lh metrics drain still fails on it.

If an ingest drain raises an exception or reports failed deliveries, the command warns on stderr and keeps its usual exit code. The outbox retains failed rows for retry.

Backoff per row: first failure waits 1 s, then doubles, capped at 300 s. The cap means an event genuinely stuck on a broken backend retries every 5 minutes forever — up to whatever [metrics].pending_ttl_days you set, after which expired rows are pruned.

End-to-end idempotency comes from the event_id field on every MetricEvent. The receiving backend is expected to upsert by it. Re-sending after a crash mid-POST is therefore safe.

Configuration¶

Minimal — sqlite_local only, no remote shipping:

# (no [metrics] block at all)

With remote shipping:

[metrics]
sinks = ["sqlite_local", "http_remote"]
user_id = "team-42"           # stamped on every event row
tenant_id = "acme"            # stamped on every event row
pending_ttl_days = 30         # outbox rows older than this are pruned

[metrics.sink_options.http_remote]
url = "https://metrics.example.com/v1/ingest"
timeout_seconds = 5.0         # per request
batch_size = 50               # events claimed per drain pass

Validation rules (enforced by the config parser):

• A name in sinks other than sqlite_local requires a matching [metrics.sink_options.<name>] block. Missing options is a hard error at load time.
• A [metrics.sink_options.<name>] block whose name is not in sinks is silently ignored (dead config).
• http_remote needs exactly one of url or url_env. Both present is a ConfigError; neither is an error when the sinks are built.
• url_env names an environment variable rather than holding the endpoint. It is read at run time, never at parse time, so the value never reaches the config file that lh rewrites. An unset or empty variable deactivates the sink: nothing is enqueued, no request is made, and the run continues on sqlite_local alone.
• If url_env is unset in the process environment, resolution falls back to <lh config dir>/secrets/metrics.env (KEY=value lines, owner-only permissions required) before deactivating the sink. This is what makes a scheduled lh metrics ingest see the same value an interactive shell does — the scheduler does not inherit the shell's exported environment. A file with group- or other-readable permissions is refused, not read. See [metrics.sink_options.<name>] in the config reference for the exact contract.

The legacy [monitoring] block (db, pricing) is independent of [metrics]. They coexist: [monitoring] controls the SQLite file's location and pricing rates for ingest cost rollups; [metrics] controls per-event fanout. A future migration may unify them; today they live side by side.

Health¶

lh metrics status always reports the local database first — session count and accumulated cost — then per-sink counts (pending / sending / sent) so a stuck http_remote is visible without inspecting the DB:

$ lh metrics status
sqlite_local  8431 sessions  $412.87  ~/.local/share/lazy-harness/metrics.db
http_remote  pending: 12  sending: 0  sent: 8431

With the default sink set (sqlite_local only), the local line is still printed and the command says no remote sinks are configured, rather than exiting silently.

For deeper inspection:

sqlite3 ~/.local/share/lazy-harness/metrics.db <<'SQL'
SELECT sink_name, COUNT(*) AS rows, SUM(attempts) AS retries
FROM sink_outbox
WHERE status = 'pending'
GROUP BY sink_name;
SQL

Pointers¶

• Pipeline: src/lazy_harness/monitoring/ingest.py
• Producer: src/lazy_harness/monitoring/collector.py
• Sink + schema: src/lazy_harness/monitoring/db.py
• Pricing table: src/lazy_harness/monitoring/pricing.py
• Sinks: src/lazy_harness/monitoring/sinks/{sqlite_local,http_remote,worker}.py
• Sink wiring: src/lazy_harness/monitoring/sink_setup.py
• Sink contracts (MetricEvent, SinkWriteResult, DrainResult, SinkHealth): src/lazy_harness/plugins/contracts.py
• CLI: src/lazy_harness/cli/metrics_cmd.py
• Tests: tests/unit/test_ingest.py, tests/unit/test_pricing.py, tests/integration/test_metrics_cmd.py