Understanding AI Agent Evals

1. What Are Evals, Really?

What Anthropic says: "Evaluations are test suites that measure agent capabilities and catch regressions before deployment."

What this means for you: Evals are like unit tests, but for AI behavior. Instead of checking if a function returns "42", you're checking if your AI agent can book a flight correctly, or if it hallucinates fake data.

Here's the structure they recommend:

• Task: A single test case (e.g., "Extract invoice total from this PDF")
• Trial: Running that task multiple times (because AI is non-deterministic)
• Grader: Logic that scores whether the agent succeeded
• Transcript: Complete record of what the agent did (tool calls, reasoning, output)

PM decision point: You need evals BEFORE launch, not after. Build them as you build the product.

2. Three Types of Graders (And When to Use Each)

This is where it gets practical. You have three ways to grade your AI agent's performance:

Code-based graders: Old-school programming checks

• Examples: String matching, checking if output is valid JSON, verifying a file was created
• Pros: Fast, cheap, deterministic, easy to debug
• Cons: Brittle—fails when AI finds a valid but different solution
• When to use: Clear right/wrong answers (e.g., "did the agent call the correct API?")

Model-based graders: Use another LLM to grade the output

• Examples: Ask Claude to score if response is "helpful and accurate on a 1-10 scale"
• Pros: Flexible, handles nuance, scales well
• Cons: Non-deterministic, costs money, needs calibration
• When to use: Subjective quality (tone, completeness, clarity)

Human graders: Actual people reviewing outputs

• Examples: Subject matter experts, crowdsourced reviews
• Pros: Gold standard, catches edge cases
• Cons: Expensive, slow, doesn't scale
• When to use: Calibrating your model-based graders, spot-checking production

PM decision point: Start with code-based graders for speed. Add model-based graders for quality. Use humans to validate both are working.

3. Capability vs Regression Evals

Capability evals: Tests where you expect to fail (for now)

• Purpose: Measure improvement over time
• Example: "Can the agent handle 10-step workflows?" (currently: 20% pass rate, goal: 80%)
• You run these to see if your changes are making the agent smarter

Regression evals: Tests you should pass every time

• Purpose: Don't break what's already working
• Example: "Can the agent still format dates correctly?" (should be: 100% pass rate)
• These are your safety net in CI/CD

Here's the lifecycle: Today's capability eval (60% pass rate, improving) becomes tomorrow's regression eval (must maintain 100%). It's like graduating a feature from beta to stable.

PM decision point: Track both. Capability evals tell you if you're getting better. Regression evals tell you if you broke something.

4. Agent-Specific Eval Strategies

Different types of agents need different eval approaches. Anthropic breaks it down:

Coding agents: Use deterministic graders

• Run the code, check if tests pass
• Verify files were created/modified correctly
• Use static analysis to check code quality
• Example benchmark: SWE-bench (GitHub issues dataset)

Conversational agents: Use simulated users + multi-dimensional success

• Have a second LLM play the customer role
• Measure: Did it solve the issue? In how many turns? Was the tone appropriate?
• Example: Customer support bot resolves return request in <5 messages with polite tone

Research agents: Combine multiple grader types

• Check: Are claims grounded in sources? Are sources high-quality? Is coverage comprehensive?
• Needs frequent calibration with domain experts
• Highly subjective—"comprehensive" means different things to different people

Computer-use agents: Run in sandboxed environments

• Give the agent a browser or OS, measure if it completes tasks
• Trade-off: Screenshots (faster) vs DOM extraction (more tokens, slower)
• Example benchmarks: WebArena, OSWorld

5. Understanding Non-Determinism Metrics

This is where AI evals differ from traditional software testing. Your agent might succeed on retry 3 but fail on retry 1. So how do you measure reliability?

pass@k: What's the chance you succeed in k attempts?

• If pass@1 = 50%, you succeed on first try half the time
• If pass@5 = 80%, you succeed within 5 tries 80% of the time
• Use this when: You can afford retries (batch jobs, background tasks)

pass^k: What's the chance all k trials succeed?

• If each trial has 75% success, then pass^3 = 0.75³ ≈ 42%
• More conservative—tells you consistency
• Use this when: You need reliability (customer-facing, real-time)

PM decision point: Know which metric matches your use case. A research tool can tolerate retries. A customer chatbot cannot.

6. Common Pitfalls (And How to Avoid Them)

Anthropic shares real mistakes teams make. Here's what to watch for:

• Over-specifying graders: Don't check every step, just the outcome. Agents might find valid approaches you didn't anticipate.
• Shared state between trials: Each test should run in a clean environment, or else the agent "learns" from previous runs and inflates scores.
• Ambiguous tasks: If two domain experts can't agree on the right answer, your eval is broken—not your agent.
• Class imbalance: Test where behavior should AND shouldn't occur. (Example: Test that agent triggers search when needed AND doesn't trigger when not needed)
• Taking scores at face value: Always read transcripts. Understand why the agent failed. Your grader might be rejecting valid solutions.
• Ignoring saturation: If you hit 100% pass rate, the eval is no longer useful for improvement. You need harder tests.