Behavioral budgets and bounded simulation investigations

These commands extend the existing runner. They do not change frozen scenario v0, proof assurance or deployment authority. The commands are subcommands of flightops-tui, which comes with the install.

Measurements and reviewed budgets

Supply a measurement sidecar when executing paired simulation revisions:

flightops-tui simulation-run --repo . --baseline BASE --candidate CANDIDATE \
  --scenario reviewed.yaml --seeds 11 29 --measurements measurements.json \
  --policy policy.json --output /tmp/new-experiment

Example measurements.json:

{
  "schema": "flightops.measurement-plan/0",
  "window_s": [0, 10],
  "max_gap_s": 0.5,
  "path": {"topic": "/odom", "frame": "map", "reference_xy": [[0, 0], [2, 0]]},
  "contacts": {"topic": "/collision_observed"}
}

Path input is nav_msgs/msg/Odometry in the exact declared frame. The extractor computes sample-weighted RMS nearest-segment XY distance; it does not infer TF or time alignment. Contact input is std_msgs/msg/Bool, interpreted as false-to-true contact onsets. Sustained true messages count once; an initially active contact leaves the onset count unknown. The producer must implement the intended contact definition. Candidate-produced odometry or Bool signals are not independent truth.

Both metrics require observations covering the requested recording-clock window within max_gap_s. Missing topics, frames, samples or explicit recording-clock t0 produce unavailable measurements. Failed trial completion times remain absent. Each extraction retains recording hash, extractor hash, definition and coverage. The worker is bounded to 128 MiB recordings, 100,000 samples, 768 MiB address space and a 30-second supervisor deadline. It never converts a collision criterion pass into a zero collision count. Clearance and latency extractors remain future work.

Example policy.json:

{
  "schema": "flightops.behavior-policy/0",
  "minimum_pairs": 2,
  "maximum_failure_rate_increase": 0,
  "budgets": {
    "completion_time_s": {"statistic": "median", "maximum_increase": 0.5},
    "collision_count": {"statistic": "max", "maximum_increase": 0}
  }
}

Budgets apply to paired deltas using mean, median, p95 or max. A missing mandatory pair is inconclusive; differing environments or seed sets are incomparable. An observed violation remains a regression even if another metric is unavailable. These are descriptive budgets, not statistical confidence or automatic deployment acceptance. The experiment command retains its existing exit semantics; inspect policy-result.json, or use the explicit policy command for a policy-gate exit:

flightops-tui behavior-policy baseline.json candidate.json \
  --policy policy.json --output new-policy-result.json

Exit codes are 0 within budget, 1 regression, 3 inconclusive/incomparable and 2 refusal.

Controlled faults

Add --faults fault.json to the simulation runner. Configure the selected launch to consume the explicitly named proxy output; the tool does not rewrite project source or intercept unrelated DDS traffic. Example:

{
  "schema": "flightops.simulation-fault/0",
  "kind": "dropout",
  "input_topic": "/raw/odom",
  "output_topic": "/flightops/fault/odom",
  "consumer_node": "/controller",
  "message_type": "nav_msgs/msg/Odometry",
  "reliability": "best_effort",
  "after_s": 3,
  "duration_s": 1,
  "enabled": true
}

dropout, delay, stale_tf and clock_pause are supported. Delay adds delay_s (at most five seconds); its queue is bounded to 256 messages and 8 MiB. Individual messages are limited to 1 MiB. Stale TF requires tf2_msgs/msg/TFMessage, parent and child; it freezes only that previously observed dynamic link with its original stamp. Other links pass through and /tf_static cannot be selected as a fault input. Clock pause uses the owned Gazebo world control service, checks clock advancement before/after and unchanged observed time during the pause, and attempts resume on cleanup. It requires at least 0.5 seconds. Check the recorded clock and application behavior before attributing a failure to a pause.

Schedules use monotonic time from helper startup, immediately before entrypoint launch; this is recorded separately from scenario simulation time. Duration is bounded to 60 seconds and onset to 300 seconds. Use enabled: false for a no-fault control. A missing expected subscriber, absent selected TF seed, incomplete window or missing applied-effect evidence invalidates the experiment. Endpoint observation is not itself proof of delivery to the application; scenario evidence remains necessary.

Revision narrowing and fault-duration reduction

flightops-tui experiment-search revisions --repo . --baseline GOOD \
  --revisions GOOD MIDDLE BAD --scenario reviewed.yaml --criterion goal \
  --seeds 11 29 --output /tmp/new-search --max-evaluations 8 --wall-budget-s 600

flightops-tui experiment-search reduce --repo . --baseline GOOD \
  --candidate BAD --faults fault.json --scenario reviewed.yaml --criterion goal \
  --seeds 11 29 --output /tmp/new-reduction --max-evaluations 8 --wall-budget-s 600

Every evaluation invokes the real build/scenario experiment runner with the same reviewed criterion and paired seeds. The reference baseline must consistently pass. Revision search tests endpoints then midpoints of an explicitly ordered commit list; it retains reverse transitions, unresolved intervals and flaky classifications. It never infers an untested globally first bad commit. Reduction tests geometrically smaller fault durations and retains the smallest verified failure found, including unsuccessful reductions. General recording/topic reduction is not implemented.

Campaigns retain search.json, every experiment tree and any smallest-found fault sidecar. A wall deadline interrupts through the existing teardown paths; interruption, unknown cleanup or incomplete evaluation stops the campaign. max-evaluations counts paired experiment executions, each containing two gates per supplied seed. Exit 0 means a narrowed result or smallest-found counterexample; 3 means incomplete, nonmonotonic or inconclusive; 2 means input refusal. All results retain assurance: none.

Explicit isolated policy evaluation

flightops-tui isolated-behavior --source small-candidate-context \
  --entrypoint main.py --baseline baseline.json --policy policy.json \
  --image LOCAL_IMAGE@sha256:EXACT_DIGEST --output /tmp/new-isolated-evaluation

The candidate is a Python entrypoint that writes a simulation behavior-runs/0 record to /candidate-output/runs.json. Its frozen context is limited to 64 files, 128 directories and 8 MiB; output is limited to 128 KiB. The image must already be local and digest-pinned. Candidate and evaluator run in separate containers with network disabled, dropped capabilities, readonly roots and no host devices/socket. The candidate has no writable host mount; its output is bounded tmpfs. A supervisor reads only a regular nonsymlink result, then the stopped candidate's record is retained before a separate evaluator starts. Source, image, supervisor, judge and evaluator code identities are retained. Both owned containers are removed afterward.

The separate evaluator mounts frozen comparator code and reviewed inputs readonly, with a distinct verdict output. This establishes a bounded evaluator write boundary for this invocation. Candidate measurement truth remains unverified: fabricated measurements can satisfy a descriptive budget. This runner does not replace the same-user Gazebo evaluator or establish independent simulator observation. Comparison assurance remains none. Exit 0 means evaluation completed; inspect the separate policy status for regression. Exit 3 means incomplete, and 2 means preflight refusal.