flightops-tui simulation-run builds two committed source revisions and executes one explicitly selected Gazebo scenario for each paired seed. It uses the existing Highbay workspace provider, gate build/test stages, workspace overlay, Gazebo adapter, recording, ROS domain leases and process teardown. It creates real runs; the separate behavior-compare command still accepts saved measurements.
The behavior and fault guide covers the new --measurements, --policy and --faults options, policy exit semantics, protected evaluation and bounded search. The package guide explains portable evidence and explicit rerun preflight. These extensions retain the existing assurance limits. Run them against your own sourced ROS and Gazebo project before relying on the results.
Run it inside a supported, sourced ROS/Gazebo environment with Git, colcon and Highbay prerequisites available. Run it from your project's repository root and pass your own scenario file with --scenario.
The example below uses the bundled scout. It needs a source checkout: it uses demo/scenarios/, which is not in the install. Run it from the repository root:
flightops-tui simulation-run \
--repo . --baseline HEAD~1 --candidate HEAD \
--scenario demo/scenarios/docking-approach-scout.yaml \
--seeds 11 29 --output /tmp/flightops-simulation-experimentSelect revisions that contain the controller being evaluated. The caller's .launchcommit.yaml supplies the common build configuration; the selected scenario supersedes its scenario list for this invocation. Relative scenario entrypoints resolve inside each revision's acquired checkout. The source checkout is not edited. An existing output directory is refused. Two to 32 distinct nonnegative seeds below 2**31 are allowed; build, test, lint and scenario timeout settings must be no more than 3,600 seconds each. Each underlying stage retains its own output and storage limits. A whole experiment may still be lengthy: every attempt executes its gate, though Highbay can reuse build caches. Lint autofixes and quarantine reruns are disabled.
The execution order alternates baseline/candidate for each seed. Each attempt starts a fresh simulator/model through the adapter; this is a recorded reset procedure, not a claim of perfect determinism. The runner requires reported simulator seed support before an attempt contributes valid comparison evidence. Without an explicit sidecar, readiness remains the adapter's model/clock checks and advisory topic-presence checks. Add --readiness readiness.json for delivered-message and lifecycle prerequisites, for example:
{
"schema": "flightops.runtime-readiness/0",
"phase": "after-entrypoint",
"timeout_s": 20,
"messages": [{"topic": "/scan", "reliability": "best_effort", "durability": "volatile"}],
"lifecycle": [{"node": "/controller", "state": "active"}]
}Use actual lifecycle nodes in your project; the bundled scout controller is not a lifecycle node, so omit that requirement for the scout. The sidecar never activates nodes. before-entrypoint is available for simulator/bridge prerequisites; after-entrypoint checks controller startup. Both use one monotonic wall-clock deadline, bounded to 120 seconds and 16 requirements of each kind. After-entrypoint waiting counts against the scenario wall timeout and retains the original entrypoint t=0. A paused ROS clock cannot stall its deadline. Message success means a message was delivered to this observer under the selected QoS, not to every application subscriber. A transient-local message may be previously published state; freshness is not inferred. The scenario's retained artifacts contain each readiness result. A missing message/state produces a refusal and runs the existing recording/teardown path. The current message probe requires complete bounded ros2 topic echo output; large image/PointCloud array dumps can exceed that output limit and be refused even when transport delivered data. This acceptance covers /scan, /odom and a small String topic, not arbitrary large sensor payloads. The normalized sidecar participates in experiment comparison identity.
The new output directory retains:
judges/original.yamland one seed-only variant per trial, with hashes.gates/: ordinary proof records, per-scenario verdicts, logs, MCAP and bundles.experiments.json: ordered attempts, source commits, common configuration, reset procedure, seed, gate result and scenario evidence, including failed attempts.baseline.jsonandcandidate.json:flightops.behavior-runs/0records.comparison.json: the existing comparator's retained-input descriptive report, usable withflightops-resultsand the saved-results terminal interfaces.
Without a measurement sidecar, only completion_time_s is derived: positive run_duration_s on the simulation clock when all selected criteria have been satisfied. It measures time to satisfy this scenario, including any criterion windows/hold times, not an inferred navigation goal time. Failed/aborted/timed-out trials have no completion-duration measurement. With --measurements, the recording extractor can additionally derive path deviation and contact-onset counts from explicitly declared topics, frames and coverage windows; see the measurement contract. Missing or insufficient observations leave those metrics unavailable. A collision criterion pass is not fabricated into a collision count. Simulation support, judge identity, recording presence and supervised teardown must be established. Required build/test gate prerequisite failures invalidate the trial.
A criterion failure remains failed; refusal, missing evidence, unseeded execution and infrastructure problems remain invalid. An interruption, unexpected exception, judge alteration or unverified cleanup stops subsequent attempts and retains them as not run/invalid. SIGTERM and keyboard interruption write an incomplete report when Python cleanup can run. SIGKILL or power loss leaves the last atomic plan showing running/pending entries; no resume/recovery execution is implemented. Never treat those entries as successful.
Runtime fingerprint differences block comparison. The deliberately distinct ROS domain IDs are excluded from that comparison; the observed backend fingerprint and hashes of available primary registry/world/model/render/bridge inputs and actual rendered world/model bytes are retained. This is not a complete dependency closure: external meshes, plugins, downloads and every transitive render input are not covered. Keep external assets fixed and inspect retained evidence before interpreting results.
The caller selects the same judge for both revisions, and its retained bytes are checked before and after execution. The seeded copies are read-only as an accidental write guard. Candidate code and the judge run as the same operating-system user; there is no independent judge write isolation or authenticated producer. A candidate could evade before/after tamper detection. Same-user protected file permissions and content hashes must not be presented as a security boundary. The report carries assurance: none; it is descriptive evidence, not automatic candidate acceptance or a hardware/deployment authorization. The runner only accepts gz-sim scenarios.
Exit 0 means the planned experiments completed with comparable valid trials; it can include behavioral failures and does not select a winner. Exit 3 means incomplete or incomparable experiments. Exit 2 means an input/setup refusal before normal completion. Individual proof statuses and every attempted outcome remain available separately.
Python API: launchcommit.experiments.run_experiments(repo=..., baseline=..., candidate=..., scenario=..., seeds=[...], output=...); integrations can dispatch launchcommit.experiments.main(argv) without introducing another record format.
Check a sourced ROS environment
The repository includes two small, non-actuating checks for a sourced ROS environment. They need a source checkout: they use scripts/, which is not in the install. Choose unused domain IDs and fresh output paths:
python scripts/simulation_readiness_acceptance.py \
--ros-domain-id 177 --output /tmp/flightops-readiness-check.json
python scripts/scout_diagnostics_acceptance.py \
--repo "$PWD" --ros-domain-id 179 \
--output /tmp/flightops-diagnostics-check.jsonThe first checks delivered messages and lifecycle states with synthetic nodes. The second checks the bundled scout controller's diagnostics with synthetic odometry and clock input. Neither checks a physical robot. The selected output files must not already exist.