The eval run object

⚠️

Preview — this feature is being built; the design below is the spec.

The eval_run object as the API returns it. Every field below appears in every response that returns this object; fields with no value are null rather than missing.

Example

{
  "id": "evrun_01JD35D7F9H1K3M5P7R9",
  "type": "eval_run",
  "name": "shorter system prompt vs baseline",
  "dataset_id": "evds_01JD13B5D7F9H1K3M5P7",
  "status": "succeeded",
  "variants": [
    {
      "label": "baseline",
      "model": "Amara S1.2",
      "prompt_version": "support-v12"
    },
    {
      "label": "short",
      "model": "Amara S1.2",
      "prompt_version": "support-v13"
    }
  ],
  "grader_ids": [
    "evgr_01JD46E8G0J2L4N6Q8S0"
  ],
  "progress": {
    "total": 400,
    "done": 400,
    "errored": 0
  },
  "summary": [
    {
      "label": "baseline",
      "pass_rate": 0.82,
      "mean_score": 4.11,
      "mean_output_tokens": 141,
      "cost_usd": "1.8410"
    },
    {
      "label": "short",
      "pass_rate": 0.86,
      "mean_score": 4.25,
      "mean_output_tokens": 98,
      "cost_usd": "1.4022"
    }
  ],
  "created_at": "2026-11-05T10:00:00Z",
  "finished_at": "2026-11-05T10:09:40Z"
}

Fields

Field	Type	Description
`id`	string	Unique id of this object.
`type`	string	The object type, always `eval_run`.
`name`	string	Name shown in the Console and in reports.
`dataset_id`	string	The dataset.
`status`	string	`queued`, `running`, `succeeded`, `failed` or `canceled`.
`variants`	array	Up to 4 prompt and model combinations, each run on every item.
`grader_ids`	array	Nested detail; see the endpoint pages below.
`progress`	object	Items done per variant summed across variants.
`summary`	array	Per variant: pass rate, mean score, mean output length and cost, once finished.
`created_at`	string	When it was created, RFC 3339 UTC (some older objects use Unix seconds).
`finished_at`	string	When it ended, or `null`.

Returned by

eval_item federation_issuer

Was this page helpful?