Compare original and candidate runs

lcc compare retains two canonical gate proofs, their declared scenario evidence, and an explicitly selected scenario in a new local directory. It compares a named failing success criterion against the candidate's results. Existing proof bytes and source recordings stay unchanged.

The first version supports saved gz-sim scenario runs. It does not execute the two revisions, evaluate a new judge, or compare standalone replay verdicts.

bash

lcc compare \
  --original /evidence/original/verdict.json \
  --candidate /evidence/candidate/verdict.json \
  --approved-scenario /reviewed-criteria/navigation.yaml \
  --criterion mission_complete \
  --output /evidence/comparisons/navigation-fix \
  --json

The output path must be new. The command never replaces an existing packet or deletes inputs. Repeat --attempt /evidence/earlier-attempt/verdict.json to retain up to 20 additional attempts. They remain visible as attempts and do not establish a repetition count or held-out acceptance result.

Argument	Meaning
`--original PATH`	Required canonical gate `verdict.json` for the failing original.
`--candidate PATH`	Required canonical gate `verdict.json` for the candidate revision.
`--approved-scenario PATH`	Required caller-selected scenario YAML, outside both recorded source workspaces.
`--criterion ID`	Required success-criterion ID in that scenario. Abort criteria cannot establish the selected original failure.
`--attempt PATH`	Optional repeatable canonical proof, up to 20 additional attempts.
`--output PATH`	Required new packet directory. Missing parent directories are created.
`--json`	Print the complete comparison record. Otherwise print status, summary and the saved record path.

The command is part of launchcommit and has no MCP operation. It accepts saved proof paths, not revision names or scenario verdicts alone. Produce the runs through the existing gate workflow first and retain their scenario artifacts.

What the result establishes

A passing comparison requires distinct resolved revisions and run IDs, matching recorded environment facts and gate configuration, and both scenario hashes matching the selected scenario's exact bytes. Per-run domain IDs, hostnames, environment-script paths and gate configuration paths may differ. The report still shows those differences.

The original must fail the selected success criterion after passing workspace, dependency, build, lint and test stages. An infrastructure error does not count as reproduction. The candidate needs passing simulation assurance and complete, passing criterion results. Both recordings and scenario verdicts must remain available under their proof directories. Missing, empty or escaping artifacts, unknown identities, incomplete results or changed criteria block comparison.

Exit codes are 0 for a recorded improvement, 1 for a still-failing candidate, 3 for an inconclusive comparison, and 2 for invalid input or publication failure. A blocked comparison retains the available evidence and explains each problem.

Status	`complete`	Exit	Interpretation
`pass`	`true`	0	The original reproduces the selected failure and the candidate passes under the recorded comparison conditions.
`fail`	`true`	1	The required evidence checks pass, but the candidate remains failing.
`blocked`	`false`	3	A prerequisite comparison check is inconclusive. Blocking checks take precedence over a candidate failure.
Input/publication refusal	No completed record	2	Invalid canonical proof/scenario, existing output, or failed publication. Read stderr.

Missing or unreadable scenario artifacts become named evidence problems when the canonical proof itself is readable. A malformed canonical proof refuses the operation; it cannot be turned into a comparison with invented identity.

Criteria and authority

--approved-scenario means the caller explicitly selected that reviewed file. It is not a signature or authenticated approval. Keep both that file and the comparison packet outside the recorded source workspaces. The command retains a copy and requires the hashes in both proofs to match it.

This command does not enforce write permissions during an earlier execution or authenticate the proof producer. Run candidates with separately enforced judge isolation when that claim is needed. A content hash establishes byte identity; it cannot establish who produced or approved those bytes. The result therefore reports runtime_isolation: unverified and describes a comparison of saved evidence. It makes no physical safety or repeatability claim.

Format and retention

The versioned flightops.comparison/0 record is comparison.json. It contains original and candidate execution/assurance separately, stage statuses, selected criterion identities/results, recorded environment/configuration and their differences, named comparison checks and relative artifact references with SHA-256 and byte length. Proofs and scenario evidence are copied into separate directories. The archived proofs retain their original bytes, including their original artifact paths; use the comparison's relative references to inspect the retained copies.

A packet has this layout. Scenario numbers follow the order in each proof, so the selected scenario need not be 001.

text

navigation-fix/
  comparison.json
  criteria/scenario.yaml
  original/proof.json
  original/scenarios/001/verdict.json
  original/scenarios/001/recording.mcap
  original/scenarios/001/console.log
  candidate/proof.json
  candidate/scenarios/001/...
  attempts/01/proof.json
  attempts/01/scenarios/001/...

Console logs and additional attempts are optional. All scenario verdicts and recordings declared by each proof are considered for retention, not only the selected scenario. Missing mandatory artifacts block original/candidate evidence. Optional console-log problems remain visible without independently blocking the comparison. Attempt problems are recorded on that attempt and do not change the original/candidate score.

Record field	Contents
`schema`, `title`, `created_at`	Version `flightops.comparison/0`, display title and creation timestamp.
`status`, `complete`, `summary`, `scope`	Comparison outcome and its limited claim.
`original`, `candidate`, `attempts`	Run ID, revision, execution, assurance, top-level proof completeness, stage statuses, recorded environment/configuration, selected criterion results, artifact references and collection problems.
`criteria`	Retained scenario reference, name, selected success ID, all criterion IDs, `approval: explicit-caller-selection`, `runtime_isolation: unverified`.
`checks`	Named comparison checks with `id`, `label`, `status`, `details` and `measured` values.
`differences`	Recorded original/candidate environment and gate-configuration differences, including allowed path/domain/hostname differences.
`warnings`	Limitations attached to this comparison, including unscored additional attempts.

Each retained reference has a packet-relative path, lowercase hexadecimal sha256 and size_bytes. Run assurance is preserved separately from execution and from comparison completeness. Inspect these fields before treating a green comparison as applicable to another environment.

flightops-results and flightops-tui consume the same comparison presentation. Rendering never opens a path supplied by an artifact reference automatically. The saved comparison records what was available when it was created; hashes allow a reviewer to check subsequent changes to retained files.

Recordings must be finalized and indexed. An isolated audit checks their summary and confirms actual messages for required topics. It does not decode sensor payloads, scan unindexed recordings, or reevaluate behavioral criteria. An unsupported recording blocks comparison while retaining its bytes. Per-audit bounds are 512 MiB memory, 5 seconds CPU, 10 seconds wall time, 64 MiB read, 8 MiB chunks/records and 10,000 inspected messages.

JSON inputs and the comparison record are capped at 4 MiB. Collection has a 120-second deadline and uses the existing LCC_MAX_ARTIFACT_BYTES, LCC_MAX_BUNDLE_BYTES and LCC_MIN_FREE_BYTES limits. Publication reserves a new directory and stages the packet before rename. Failed publication preserves source evidence. Completed packets have no automatic expiration or deletion.

Bound	Default
`LCC_MAX_ARTIFACT_BYTES`	2 GiB per copied scenario artifact.
`LCC_MAX_BUNDLE_BYTES`	4 GiB for the retained packet.
`LCC_MIN_FREE_BYTES`	512 MiB free-space reserve.
Scenarios per proof	100.
Additional attempt proofs	20.
MCAP summary	4 MiB and 4,096 summary records.
MCAP channel/chunk indexes	1,024 channels and 4,096 chunk indexes.
Required recording topics	128, each at most 512 characters.

The three environment variables accept integer byte counts. Artifact/bundle caps must be positive and the free-space reserve nonnegative. Raising storage budgets does not raise the recording audit's CPU, memory, index or read limits. The packet copies full recordings; it is not a bounded evidence-window query.

Python and saved-result APIs

python

import json
from pathlib import Path
from launchcommit.comparison import compare, validate_comparison

record_path = compare(
    Path("/evidence/original/verdict.json"),
    Path("/evidence/candidate/verdict.json"),
    approved_scenario=Path("/reviewed-criteria/navigation.yaml"),
    criterion="mission_complete",
    output=Path("/evidence/comparisons/navigation-fix"),
    attempts=(),
)
record = json.loads(record_path.read_text())
validate_comparison(record)

compare returns the path to comparison.json, including for a published blocked or fail result. Inspect its status; successful return means publication completed. validate_comparison checks the schema, named checks and derived comparison semantics. It does not open retained artifact paths or authenticate the original executions. presentation(record, source_path) validates the record and returns the shared HTML/TUI model.

bash

flightops-results /evidence/comparisons/navigation-fix/comparison.json \
  -o /evidence/comparisons/navigation-fix.html
flightops-tui --plain /evidence/comparisons/navigation-fix/comparison.json

The HTML labels this a comparison result and shows original execution separately from candidate execution. Its green comparison status does not replace either run's recorded result.

Troubleshooting and verification

Check or symptom	Interpretation and next step
`source-identities`	Both proofs need distinct resolved commit IDs and distinct run IDs. Reusing one run as both sides cannot demonstrate a change.
`selected-criteria`	Both proof hashes must equal the exact reviewed YAML bytes. A formatting change also changes that hash.
`criteria-location`	Put selected criteria and output outside the source/workspace roots recorded in the proofs. This is a location check, not historical write isolation.
`environment` or `configuration`	Review the explicit differences. Run comparable conditions instead of editing old proof fields.
`original-evidence` or `candidate-evidence`	Restore the original referenced artifacts under their proof directory, or rerun to obtain complete evidence. Do not replace missing data with empty files.
`original-failure`	The selected success criterion must be `FAIL`; workspace, dependency, build, lint and test stages must have passed. Infrastructure failure is inconclusive.
`candidate-result`	A green gate alone is insufficient. The candidate needs passing `prove-sim` assurance and passing selected-scenario criteria.
Recording audit refuses an index, chunk or timeout	Use a finalized supported recording. The audit will not scan unindexed files or silently accept oversized chunks.

Focused regressions need a source checkout: they are in launchcommit/tests/, which is not in the install. From the repository root:

bash

.venv/bin/python -m pytest launchcommit/tests/test_comparison.py -q

Tests cover pass/fail/blocked combinations, contradictory proofs, unavailable or escaping artifacts, malformed/empty recordings, required-topic checks, criteria changes and publication/storage refusal. Generated proof fixtures exercise the comparison implementation; they are not evidence that a real robot fix was reproduced on another machine.