indiefounder
About

How to Build a Production Evaluation Suite for LLM Applications and AI Agents

A practical engineering framework for testing generative AI products: automated golden datasets, semantic assertions, LLM-as-a-Judge rubrics, and CI/CD regression gates.

By IndieFounder Editorial Team

The most common failure mode in generative AI startups is the “vibe check”: a developer edits a system prompt, runs three queries in the browser, confirms that the output looks reasonable, and deploys directly to production.

Two days later, customers complain that the model hallucinated invalid URLs, broke structured JSON output, or began leaking internal prompt directives.

Unlike traditional deterministic software, where unit tests check exact equality (assert add(2, 2) == 4), large language models (LLMs) are probabilistic and non-deterministic. Shipping reliable AI products requires a continuous evaluation (evals) pipeline that catches regressions before your users encounter them.

Continuous Evaluation Pipeline for Production LLM Applications
Figure 1: Automated CI/CD evaluation architecture testing prompt revisions against golden datasets.

The Three Tiers of LLM Evaluation

A production eval suite combines three complementary layers of testing:

Eval Layer Execution Speed Cost Purpose
Deterministic Code Assertions Sub-millisecond Free JSON schema validity, required key presence, regex format, latency limits
Semantic Distance Metrics ~10–50ms Inexpensive Cosine similarity between embeddings, BLEU / ROUGE scoring
Model-as-a-Judge (LLM Judge) ~1–3s Moderate (API calls) Evaluating tone, factual consistency, safety guidelines, and hallucination

Constructing Your “Golden Dataset”

You cannot build an eval suite without a reference test suite. The foundation of your pipeline is a Golden Dataset: a curated collection of 50 to 200 real-world input prompts paired with expected outputs or evaluation criteria.

Golden Dataset Structure (test_cases.jsonl)

{
  "id": "tc_financial_summary_01",
  "input": "Summarize Q3 earnings: Revenue $4.2M (+14%), Churn 2.1%, OpEx $3.1M.",
  "expected_keys": ["revenue", "growth_rate", "churn", "operating_profit"],
  "forbidden_substrings": ["NaN", "undefined", "profit margin of 50%"],
  "evaluation_criteria": "The response must accurately calculate operating profit as $1.1M ($4.2M - $3.1M) without guessing unstated numbers."
}

Populate your dataset with:

  • Historical failure cases: Every bug reported by a user in production becomes a permanent test case in your suite.
  • Edge cases: Inputs with missing context, non-English text, extreme token lengths, and deliberate prompt injection attempts.
  • Representative queries: Typical everyday requests reflecting core user behavior.

Implementing Deterministic and Model Judges

A minimal, robust evaluation script can be written in less than 80 lines of TypeScript or Python using standard test runners like Vitest or Pytest:

// run-evals.ts: Execute automated prompt regression test
import { test, expect } from 'vitest';
import { generateProductSummary } from '../src/agent';
import { evaluateWithJudgeModel } from './judge';

test('Financial summary agent maintains mathematical fidelity', async () => {
  const input = "Q3 Results: Revenue $10M, COGS $3M, Marketing $2M, Net Income $5M.";
  
  const startTime = Date.now();
  const response = await generateProductSummary(input);
  const durationMs = Date.now() - startTime;

  // Tier 1: Deterministic checks
  expect(durationMs).toBeLessThan(3500); // Latency constraint
  expect(() => JSON.parse(response)).not.toThrow(); // JSON structure constraint
  
  const parsed = JSON.parse(response);
  expect(parsed.gross_profit).toBe(7000000); // 10M - 3M
  
  // Tier 2: Model-as-a-Judge evaluation for hallucination
  const judgeScore = await evaluateWithJudgeModel({
    prompt: input,
    output: response,
    rubric: "Score from 1 to 5 based on whether the output invents any metric not stated in the source text. 5 = zero hallucinations."
  });
  
  expect(judgeScore).toBeGreaterThanOrEqual(4);
});

Running Evals in CI/CD Pull Requests

Integrate your eval suite into your GitHub Actions workflow. Whenever a developer modifies a prompt template or upgrades a base model version (e.g., from gpt-4o-mini to a newer checkpoint):

  1. The CI runner executes the test suite across your Golden Dataset.
  2. It generates a diff summary comparing current pass rates against the main branch.
  3. If the overall accuracy score drops below a pre-set threshold (e.g., 94%), or if any critical safety assertion fails, the pull request is automatically blocked from merging.
# .github/workflows/evals.yml
name: Prompt Regression Evals
on: [pull_request]

jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - name: Run Model Benchmark Suite
        env:
          OPENAI_API_KEY: $
        run: npm run test:evals

By substituting subjective guessing with automated test harnesses, indie founders can iterate on complex AI workflows with the same velocity and confidence as traditional software engineering.

Keep exploring

Related reading

Browse all ›