Bounded context reads and duplicate discovery

Tracked by pm-pshhry, pm-ydshl9, pm-bab3gb, pm-fx80w2, pm-gtw5zh, and pm-s8ybl9.

Project management is context management. Routine reads should pay for the requested evidence, and derived feedback should not grow with every candidate in a project's history. These primitives share SDK implementations across CLI, MCP, and package consumers.

Point reads

pm get <id> at standard depth reads the addressed item and its own history. It does not enumerate unrelated items to compute child counts. --depth brief and --fields id,title also omit claim-history work and heavy output facets. The cost still depends on the addressed record, registered type directories, and extension activation; this is not a constant-latency guarantee for arbitrary extensions or unbounded individual history streams.

Request hierarchy work explicitly with --fields id,children, --tree, or a deep/full container read. These paths preserve registered hierarchy semantics, including inverse and custom relationship kinds, rather than treating only the scalar parent field as authoritative. An omitted children facet carries a restore selector in the omission receipt. Explicit deep/full and child-field projections retain a zero-count rollup for an empty container.

Caller-carried output sessions retain only portable item identities in next_state.seen_item_ids. Workspace activity such as _workspace remains visible in full on subsequent reads. It cannot become an invalid item reference or make the next read reject a state produced by the previous read.

The optional SQLite projection binds each FTS document to the corresponding items.rowid. Maintenance resolves the indexed item identity and deletes the single FTS row, avoiding a scan of the unindexed FTS id column on every insert, update, or deletion. Derived index version 5 rejects older projections and rebuilds them from authoritative metadata. Real SQLite query-plan tests retain the old predicate as a negative control and verify search results after updates and deletions. SQLite documents the distinction between FTS columns and row identity.

Exact duplicate candidates

pm duplicates --status all --threshold 0.8 --json
pm duplicates --status open --exhaustive --json

The default algorithm orders title tokens by corpus frequency and uses a Jaccard prefix join. For a set of size n and threshold t, a prefix of length n - ceil(t*n) + 1 must intersect the similarly ordered prefix of every qualifying set. The length filter rejects impossible token matches before scoring. Exact normalized titles, issue codes, and empty-token matches retain the canonical scorer's separate rules. This follows the established all-pairs similarity-search approach.

Both modes retain exact recall relative to the canonical title scorer. cost discloses algorithm, item_count, possible_pairs, candidate_pairs, scored_pairs, pair_limit, and recall_guarantee. The one-million-candidate safety ceiling remains enforced. Dense duplicate output or a zero threshold can still require quadratic work; the command refuses without returning a partial answer. Exact recall is not a claim that every semantic duplicate has similar titles. Review the actual item evidence before merging or closing.

Packages with remote or custom stores can use the same kernel without building a filesystem tracker:

import { analyzeDuplicateItems } from "@unbrained/pm-cli/sdk/query";

const result = analyzeDuplicateItems([
  { id: "work-a", title: "Publish research protocol", type: "Task", status: "open" },
  { id: "work-b", title: "Publish research protocol", type: "Task", status: "closed" },
]);
console.log(result.clusters, result.cost);

The function preserves its input, validates unique nonempty identifiers, and returns deterministic components and scored pair evidence. The filesystem findDuplicateClusters, PmClient.duplicates, CLI and MCP adapters reuse it. Record the algorithm and cost receipt in create-time duplicate-check evidence.

Feedback storage

Serving receipts retain the full candidate population in memory for final delivery validation. Persisted events retain only the first 256 candidate rows, with explicit population and omission counts. Only sampled ids observed in the final emitted result can train affinity. This deterministic sample has no unbiased-propensity claim. Unknown row properties are not persisted.

CONTEXT_USAGE_LIMITS publishes a 256 KiB physical ceiling, a 32 KiB individual event ceiling, 2,048 retained events and a 30-day default horizon. Writers share a cross-process lock. High-water compaction atomically replaces the ledger with a suffix of at most half the byte ceiling, leaving append headroom. Readers consume at most the byte ceiling even for an oversized legacy file. Custom retention controls can tighten the limits; invalid controls fail before I/O.

Serving receipt storage reports actual written_bytes, ledger_bytes, compacted, and lock_wait_ms. Written bytes count ledger data rather than lock metadata or filesystem block allocation. The receipt is attached to results through a non-JSON symbol and does not inflate normal CLI output. Storage is derived and disposable; item history remains independent. Ledger access refuses existing symbolic links, shared hard links, non-regular files, and a redirected runtime directory. Appends use a validated descriptor with no-follow opening where supported; replacement files are created exclusively with private permissions. Compaction closes both data handles before replacing the ledger, including on Windows.

Feedback persistence requires trusted, stable workspace directory entries and ancestors. It is unsafe when an untrusted actor can concurrently replace those entries: portable Node filesystem APIs do not pin ancestor directories through the whole operation. The link checks reject existing redirections; the lock coordinates cooperating SDK writers and is not a security boundary. Before using an adversary-writable live workspace, set PM_CONTEXT_USAGE_DISABLED=1 to disable feedback reads and writes, or work in an isolated copy owned by the caller. Public feedback API callers can also pass enabled: false.

Reproducible measurements

pnpm build
node scripts/bench/context-read-costs.mjs 10000 100000 1000000
node scripts/bench/point-read-costs.mjs 100 10000 100000

The script uses paired titles with common project vocabulary, compares a 1,000-item subset against exhaustive scoring, and runs four concurrent writers for 40 serves per tier in temporary storage. It asserts exact reference equality, expected pair counts, reachable byte ceilings, and append headroom. Property tests additionally vary thresholds, Unicode, punctuation, and issue identifiers.

The point-read script holds the addressed Epic constant while varying corpus size. The public measureItemMetadataReadWork query SDK wrapper reports actual item-store enumeration calls and returned metadata rows, including cache hits. Ordinary reads must perform zero enumerations; an explicit child scan must observe every generated item. A regression test deliberately enumerates and discards the corpus before a normal read, proving that output shape alone cannot satisfy this work gate. Measurements preserve operation failures, isolate concurrent roots, include nested work, and release their observers on completion. They cover item-store enumeration APIs rather than arbitrary extension I/O. Fixture setup registers all generated types, including Story, through the live SDK schema and uses deterministic SDK execution for byte-reproducible workspace history.

The 2026-09-05 filesystem run observed the following timings (10 reads per projection, including feedback writes). The explicit scan includes the cold derived-index rebuild; these are host observations rather than latency gates.

Measured on 2026-09-05 with Node 26.7.0; these are observations, not portable latency guarantees. Duplicate timings exclude filesystem ingestion.

The same four-writer runs include full in-memory candidate validation and final delivery recording. Throughput therefore still falls with candidate population.

Lock-wait percentiles use the empirical nearest rank across all 40 serving receipts; these are four concurrent tasks sharing a cross-process lock, not a measurement of four separate OS processes.

Reference recall was 100% at every tier. The million-item batch does not meet the broader one-second project target; persistent incremental ingestion and analysis remain necessary for that target. Full candidate validation still costs O(candidate count) in memory even though persisted feedback is bounded. On the live 2,638-item corpus, default discovery completed after scoring 3,136 of 3,478,203 possible pairs; the prior algorithm refused at its safety ceiling.