Documentation

Startup benchmark runner qualification

Tracked by pm-bj7rq0.

The SDK import and CLI transport gates retain their absolute latency and RSS ratchets. A bare Node control determines whether the runner qualifies independently of product admission. Both benchmarks bracket their census with post-warmup controls before and after it; the CLI uses five samples per control. Both use the existing SDK bare-process budget and its fixed 30 ms scheduler margin.

Successful checks explicitly return runner_qualification.status=qualified and product_admission=passed. Failed checks retain the complete measured report, every product violation, runtime identity, available parallelism, load averages, and memory observations. benchmark_runner_unqualified leaves product admission unverified; it still exits unsuccessfully. A qualified runner with an over-budget command returns benchmark_product_budget_exceeded and failed admission.

If the final control process fails after product measurements, its unavailable status and error remain in baseline_after; the completed report and every product violation remain available. Check mode rejects admission as unverified, and update mode refuses to write a calibration from that failed control. Calibration also requires both available controls to qualify against the existing SDK control budget before any budget or documentation write. A new SDK calibration target uses the committed control budget; an existing target retains its own prior control ceiling for admission. An unqualified runner cannot raise its own ceiling to certify its measurements. Even qualified controls cannot increase the prior control ceiling: update mode compares the proposed p95-derived ceiling with the existing ceiling and refuses an increase before creating directories or writing calibration files. Equal or reduced ceilings remain admissible. This refusal returns benchmark_control_budget_increase with product admission unverified; a qualified control does not establish a measured product failure.

Control cost is never subtracted from product measurements. Pressure observations are diagnostic context and do not prove the cause of a failure. An unchanged base comparison on the same runtime helps distinguish a new regression from a runner that cannot qualify; a failed or unavailable comparison cannot certify a pass.

From docs/performance/runner-qualification.md in pm 2026.10.13. View on GitHub