IndieFounder — AI & automation

How to Build a Production Evaluation Suite for LLM Applications and AI Agents

2026-10-06T09:45:00+05:30

The most common failure mode in generative AI startups is the “vibe check”: a developer edits a system prompt, runs three queries in the browser, confirms that the output looks reasonable, and deploys directly to production.

Two days later, customers complain that the model hallucinated invalid URLs, broke structured JSON output, or began leaking internal prompt directives.

Unlike traditional deterministic software, where unit tests check exact equality (assert add(2, 2) == 4), large language models (LLMs) are probabilistic and non-deterministic. Shipping reliable AI products requires a continuous evaluation (evals) pipeline that catches regressions before your users encounter them.

Figure 1: Automated CI/CD evaluation architecture testing prompt revisions against golden datasets.

The Three Tiers of LLM Evaluation

A production eval suite combines three complementary layers of testing:

Eval Layer	Execution Speed	Cost	Purpose
Deterministic Code Assertions	Sub-millisecond	Free	JSON schema validity, required key presence, regex format, latency limits
Semantic Distance Metrics	~10–50ms	Inexpensive	Cosine similarity between embeddings, BLEU / ROUGE scoring
Model-as-a-Judge (LLM Judge)	~1–3s	Moderate (API calls)	Evaluating tone, factual consistency, safety guidelines, and hallucination

Constructing Your “Golden Dataset”

You cannot build an eval suite without a reference test suite. The foundation of your pipeline is a Golden Dataset: a curated collection of 50 to 200 real-world input prompts paired with expected outputs or evaluation criteria.

Golden Dataset Structure (`test_cases.jsonl`)

{
  "id": "tc_financial_summary_01",
  "input": "Summarize Q3 earnings: Revenue $4.2M (+14%), Churn 2.1%, OpEx $3.1M.",
  "expected_keys": ["revenue", "growth_rate", "churn", "operating_profit"],
  "forbidden_substrings": ["NaN", "undefined", "profit margin of 50%"],
  "evaluation_criteria": "The response must accurately calculate operating profit as $1.1M ($4.2M - $3.1M) without guessing unstated numbers."
}

Populate your dataset with:

Historical failure cases: Every bug reported by a user in production becomes a permanent test case in your suite.
Edge cases: Inputs with missing context, non-English text, extreme token lengths, and deliberate prompt injection attempts.
Representative queries: Typical everyday requests reflecting core user behavior.

Implementing Deterministic and Model Judges

A minimal, robust evaluation script can be written in less than 80 lines of TypeScript or Python using standard test runners like Vitest or Pytest:

// run-evals.ts: Execute automated prompt regression test
import { test, expect } from 'vitest';
import { generateProductSummary } from '../src/agent';
import { evaluateWithJudgeModel } from './judge';

test('Financial summary agent maintains mathematical fidelity', async () => {
  const input = "Q3 Results: Revenue $10M, COGS $3M, Marketing $2M, Net Income $5M.";
  
  const startTime = Date.now();
  const response = await generateProductSummary(input);
  const durationMs = Date.now() - startTime;

  // Tier 1: Deterministic checks
  expect(durationMs).toBeLessThan(3500); // Latency constraint
  expect(() => JSON.parse(response)).not.toThrow(); // JSON structure constraint
  
  const parsed = JSON.parse(response);
  expect(parsed.gross_profit).toBe(7000000); // 10M - 3M
  
  // Tier 2: Model-as-a-Judge evaluation for hallucination
  const judgeScore = await evaluateWithJudgeModel({
    prompt: input,
    output: response,
    rubric: "Score from 1 to 5 based on whether the output invents any metric not stated in the source text. 5 = zero hallucinations."
  });
  
  expect(judgeScore).toBeGreaterThanOrEqual(4);
});

Running Evals in CI/CD Pull Requests

Integrate your eval suite into your GitHub Actions workflow. Whenever a developer modifies a prompt template or upgrades a base model version (e.g., from gpt-4o-mini to a newer checkpoint):

The CI runner executes the test suite across your Golden Dataset.
It generates a diff summary comparing current pass rates against the main branch.
If the overall accuracy score drops below a pre-set threshold (e.g., 94%), or if any critical safety assertion fails, the pull request is automatically blocked from merging.

# .github/workflows/evals.yml
name: Prompt Regression Evals
on: [pull_request]

jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - name: Run Model Benchmark Suite
        env:
          OPENAI_API_KEY: $
        run: npm run test:evals

By substituting subjective guessing with automated test harnesses, indie founders can iterate on complex AI workflows with the same velocity and confidence as traditional software engineering.

When Should You Use an AI Agent in a Product?

2026-10-03T10:15:00+05:30

AI agents are useful when software must interpret changing context, choose among actions, and work through a task that cannot be fully described as a fixed sequence of rules. They are not automatically the best way to add intelligence to an application. In many products, a conventional function or a single model call is cheaper, easier to test, and more reliable.

The design question is not “Can an agent do this?” It is “Does the flexibility of an agent solve a real problem that simpler software cannot solve well enough?”

For a new AI product, this technical decision should follow evidence that the customer problem matters. Use our SaaS idea validation process to test demand before investing in a complex agent workflow.

Understand the three patterns

Deterministic software follows explicit rules. A payment total, permission check, subscription status, or inventory update should usually be implemented with ordinary code and validated data. The behavior is predictable and can be tested against defined expectations.

A single model call is useful when a bounded task needs language understanding or generation. Examples include classifying a support message, summarizing a document, or drafting a response from supplied context. The application decides what happens before and after the model call.

An agent can select tools, gather additional information, decide which step to take next, and repeat until a stopping condition is reached. For example, a research assistant might search an approved knowledge base, open relevant documents, compare findings, and produce a sourced summary. This flexibility comes with additional failure modes.

These patterns can coexist. A well-designed product often uses deterministic code for sensitive state changes, a model for interpretation, and an agent only for the part that benefits from adaptive multi-step work.

Look for the right problem characteristics

An agent becomes more attractive when the task has multiple meaningful steps and the next step depends on what the previous step reveals. The required information may not be available at the start, and the system may need to choose from a small, well-defined set of tools. Mistakes must be detectable or containable, and the value of completing the task must justify added latency and cost.

Consider internal support triage. A basic classifier may assign a category. A more capable workflow might inspect a ticket, retrieve approved documentation, identify missing details, and draft a response for a human reviewer. If the task is simply to label one of five known categories, an agent may be unnecessary. If it must gather context from several approved sources, adaptive steps may provide value.

Calculate the cost of flexibility

Every extra model call or tool invocation adds latency, cost, and another opportunity for failure. An agent may also require tracing, evaluation datasets, permission boundaries, and ongoing maintenance.

Before implementation, estimate model and tool cost per completed task, typical and worst-case execution time, the rate of human intervention, the cost of a wrong action, and the effort required to monitor failures. Compare the agent with the best practical alternative, including deterministic automation and a single model call. A slow agent that occasionally produces impressive results may be worse than a fast workflow that handles common cases consistently.

Design tools as security boundaries

An agent should not receive broad access merely because it might need it. Give it the smallest set of tools and permissions required. Separate read operations from write operations, validate tool inputs, and enforce authorization in the application layer.

For example, a model may propose issuing a refund, but trusted application code should verify the user’s permissions, order state, refund limits, and business rules before a transaction occurs. Never treat a model’s claim that a user is an administrator as proof of authorization. Untrusted document text or retrieved webpages must not override application policies.

High-impact actions should require explicit confirmation or human approval. Log tool calls and outcomes without unnecessarily recording sensitive data. Use timeouts, rate limits, bounded retries, and a clear maximum step count.

Make failures visible and recoverable

Agent workflows can fail in ways ordinary functions often do not. A tool may return malformed data. A model may choose the wrong tool, repeat a step, misinterpret an instruction, or claim that work succeeded when it did not. External services may time out.

Validate structured outputs against schemas. Treat tool responses as untrusted input. Confirm that a requested operation actually succeeded before reporting completion. Add idempotency controls to prevent duplicate side effects and define a fallback when a dependency is unavailable.

A useful workflow state machine might include pending, running, awaiting approval, completed, failed, and cancelled. Do not make an agent the sole source of truth for payments, permissions, orders, or other critical business state.

Evaluate outcomes, not impressive demos

A demo usually shows favorable examples. A product needs repeatable evidence. Create an evaluation set based on representative tasks, including ambiguous requests, missing information, tool errors, malicious instructions in retrieved content, and cases where the correct behavior is to ask for clarification.

Track task success, factual correctness, tool selection accuracy, policy violations, completion time, cost, and human correction rate. Inspect failure examples rather than relying only on an aggregate score. Run evaluations before launch and continue monitoring after release because model and tool changes can alter behavior.

Start with a bounded agent

Choose one narrow workflow, a small tool set, and a clear completion condition. Keep high-impact actions behind approval. Give the workflow an explicit budget for steps, time, and model usage.

A practical workflow accepts a scoped task, gathers relevant context, selects from approved tools, validates each tool request and response, and stops when the task is complete, unsafe, or out of budget. It should present evidence and request approval where needed. This structure is easier to understand, test, and improve than a system that promises to do everything.

A practical decision checklist

Use deterministic code when rules are stable and outcomes must be exact. Use a single model call when the task is bounded and mostly involves interpreting or generating content. Consider an agent when adaptive multi-step work is central, tool use adds measurable value, and failures can be managed safely.

Before shipping, ask what user problem the agent solves better than a simpler alternative, what tools it can access, how wrong results will be detected, which actions require approval, what the cost and stopping limits are, and how quality will be measured after release. If these questions have no clear answers, the product probably needs a clearer workflow before it needs an agent.

The takeaway

AI agents are a design choice, not a product requirement. Use them where uncertainty and multi-step decisions are central, and keep predictable business logic in ordinary software. A bounded agent with narrow permissions, measurable outcomes, and safe failure handling is more useful than a broadly autonomous system that cannot reliably verify its actions.