Module 4: Evals & Reliability

Golden Datasets & Test Scenarios

How to curate, version, and synthetically augment high-signal benchmark datasets for agent evaluation.

You cannot improve what you do not measure. A golden dataset is a curated collection of real-world user prompts, mock environment states, and expected tool-call trajectories.

Anatomy of an Evaluation Case

Store evaluation benchmarks in version-controlled JSON files:

{
  "id": "eval-db-042",
  "category": "sql_generation",
  "input_prompt": "Show me our top 5 enterprise accounts by revenue in 2025 who churned in Q3.",
  "mock_environment": {
    "tables": ["accounts", "subscriptions", "invoices"]
  },
  "expected_trajectory": [
    {
      "tool": "inspect_schema",
      "args": {"tables": ["accounts", "subscriptions"]}
    },
    {
      "tool": "execute_query",
      "args_contain": ["SELECT", "WHERE status = 'churned'", "LIMIT 5"]
    }
  ],
  "pass_criteria": {
    "max_turns": 4,
    "forbidden_tools": ["drop_table", "truncate_table"],
    "required_metrics": ["churned_arr"]
  }
}

Collecting Production Edge Cases

Whenever a user submits negative feedback (e.g. thumbs down or support ticket):

1. Export the full prompt and tool trajectory.
2. Scrub sensitive personal identifiable information (PII).
3. Commit it to tests/evals/regression/ with a failing test assertion.

Continue to CI/CD Regression Testing.

Was this page helpful?