ak test benchmark

Use ak test benchmark for the operation described below. The command help records its arguments, effects, and options.

Use only a corpus you trust: the command executes its commands on your machine.

Read a comparison

Each corpus case runs in fresh temporary workspaces with isolated tool homes. The corpus declares source fixtures, changed paths, the manifest, policy, and the independent set of groups expected to fail. Runs alternate baseline and selection order and record the actual commands, exit statuses, elapsed time, attempts, and selected or skipped reasons.

The full baseline must detect exactly the expected failures. A setup error, a timeout, a missing required check, or a wrong oracle makes that trial invalid, and the command exits 1. An expected regression must exit 1 unless the corpus declares another code from 1 to 125 for that group in expected_exit_codes; both baseline and selection must match it exactly, and a command-not-found or signal outcome never satisfies the oracle. Corpus commands should separate setup failures from assertion failures, because the runner cannot tell apart two causes that return the same expected code.

A successful comparison reports zero missed known regressions in a synthetic corpus. It is not evidence of universal coverage or production savings, and cost stays null when nothing measured it. --output follows the same new-file-only rule as ak test optimize.

Command contract

Execute trusted regression corpus in isolated temporary workspaces

Synopsis

Execute a trusted corpus in fresh temporary workspaces with isolated tool homes. This is not an OS sandbox. Compare repeated baseline and selected checks against an independent failure oracle. Measurements describe synthetic regression detection, not production savings.

Effects: Executes corpus commands, creates and removes temporary workspaces, and optionally creates a local JSON receipt without overwriting an existing file.

Exit status: 0 all comparison trials valid with no missed regressions 1 invalid or failed comparison, or input/output failure 2 invalid arguments

ak test benchmark <corpus.json> [flags]

Examples

  ak test benchmark corpus.json --json
  ak test benchmark corpus.json --output comparison.json

Options

      --e2e-labels string        Independent E2E labels for a replay containing classifier advice
  -h, --help                     help for benchmark
      --json                     Emit machine-readable JSON (implies --no-interactive)
      --no-interactive           Disable interactive prompts (CI-safe)
      --output string            Create a new JSON comparison receipt; never overwrite
  -q, --quiet                    Suppress non-error output on stderr
      --semantic-replay string   Replay frozen semantic observations offline; never call a provider
  -V, --verbose                  Extra diagnostic output on stderr (loses to --quiet)
  -y, --yes                      Assume yes for all prompts

SEE ALSO

• ak test - Plan and execute repository test-impact checks