ak eval run

Use ak eval run for the operation described below. The command help records its arguments, effects, and options.

Command contract

Execute repeated isolated evaluation trials

Synopsis

Execute the suite's commands and graders. This is workspace isolation, not an OS sandbox; use only trusted suites.

Effects: Runs configured local commands, possibly paid model calls; writes local measurements and removes temporary workspaces.

Exit status: 0 suite finished; inspect failure counts 1 suite or measurement error 2 invalid arguments

ak eval run <suite.json> [flags]

Examples

  ak eval run suite.json --json
  ak eval run holdout-suite.json

Options

  -h, --help             help for run
      --json             Emit machine-readable JSON (implies --no-interactive)
      --no-interactive   Disable interactive prompts (CI-safe)
  -q, --quiet            Suppress non-error output on stderr
  -V, --verbose          Extra diagnostic output on stderr (loses to --quiet)
  -y, --yes              Assume yes for all prompts

SEE ALSO

• ak eval - Run controlled skill and agent evaluations