Offline evidence selection comparison

flightops_tools.ai.jev_evaluation compares saved evidence-selection results for three routes:

1. Pass every candidate to Codex without a selector. This is a deterministic all-candidates control, not a measure of Codex's ability to select evidence.
2. Ask a small model to make one constrained priority recommendation, then continue with Codex.
3. Ask Jev to make one constrained priority recommendation, then continue with Codex.

The command only reads a JSON file and prints a JSON report. It does not call Jev, Codex, or any other provider:

flightops ai compare-evidence trial.json > report.json

The input has schema flightops.ai-evaluation-data/1. Each incident supplies a stable ID, the candidate evidence IDs offered to every route, human-reviewed IDs judged relevant to the selection question, and one observation for each route. expected_abstention is true exactly when no offered candidate is relevant. Each incident has 1–16 candidates. Candidate IDs are unique within an incident, relevance IDs must be a subset of the candidates, and selector outputs must be candidate IDs. The baseline must select the complete candidate set. Model selector routes use one choice or abstention, matching Jev's Choice interface. All three route records are required for every incident; the evaluator refuses partial route coverage instead of reporting a misleading comparison.

The values below are synthetic examples only. Do not copy them as measured latency, token use, or cost.

{
  "schema": "flightops.ai-evaluation-data/1",
  "incidents": [
    {
      "id": "navigation-gap-01",
      "candidate_ids": ["receipt-run-01", "receipt-check-01"],
      "relevant_ids": ["receipt-check-01"],
      "expected_abstention": false,
      "observations": {
        "baseline_no_selector_codex": {
          "selected_ids": ["receipt-run-01", "receipt-check-01"],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        },
        "small_model_plus_codex": {
          "selected_ids": ["receipt-check-01"],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        },
        "jev_plus_codex": {
          "selected_ids": ["receipt-check-01"],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        }
      }
    },
    {
      "id": "driver-status-unavailable-01",
      "candidate_ids": ["receipt-recording-02", "receipt-check-02"],
      "relevant_ids": [],
      "expected_abstention": true,
      "observations": {
        "baseline_no_selector_codex": {
          "selected_ids": ["receipt-recording-02", "receipt-check-02"],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        },
        "small_model_plus_codex": {
          "selected_ids": [],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        },
        "jev_plus_codex": {
          "selected_ids": [],
          "input_tokens": null,
          "output_tokens": null,
          "elapsed_seconds": null,
          "cost_usd": null
        }
      }
    }
  ]
}

Use the same incident, evidence packet, candidate IDs, relevance labels, and selection question for both selector routes. For the small-model route, present the same bounded receipts and ask it to choose exactly one supplied ID or insufficient_evidence; accept only an ID from that enum. For Jev, record its validated choice: map insufficient_evidence to selected_ids: [], otherwise use [selected_receipt_id]. These selected_ids are selector recommendations that the harness scores. In the current assistant path, Jev marks a priority while all user-attached receipts remain in Codex context; this report therefore does not claim that Jev reduced the evidence sent to Codex. Record the IDs actually attached to Codex separately in your experiment notes. A future evidence-dropping mode would need a separate evaluation of context reduction and downstream answer quality. A Jev probability or a Codex explanation does not establish that a diagnosis is true.

Enter recorded token counts and cost for the full route: selector plus Codex for the two selection routes, and Codex alone for the baseline. Enter measured end-to-end time from the start of selection through the final Codex response. Do not estimate or guess timing or charges. Use null for any value that was not measured; do not enter zero as a placeholder. The report's exact_total is null if any incident measurement is unknown. observed_subtotal reports only the known values and is not a complete total when unknown_incidents is nonzero. It is null when no incidents have a recorded value.

The report includes micro evidence precision and recall, the fraction of incidents with a nonempty selection, evidence coverage among answerable incidents, correct abstention, and per-incident selection outcomes. These metrics measure agreement with the supplied relevance labels. They do not measure root-cause accuracy, causal reasoning, downstream explanation quality, or safety. Label incidents independently of the model outputs, split future evaluation by whole incident or project, and avoid presenting a small synthetic sample as a benchmark.