---
title: "Evaluating and improving an AI app, explained with an exam"
description: "build-eval writes an exam for your AI app with you; hillclimb revises against it one change at a time, with held-out cases to catch cramming."
canonical: "https://me.consonance.fyi/en/blog/automating-eval-design-and-hillclimbing/"
lang: "en"
published: "2026-10-03"
---

# Evaluating and improving an AI app, explained with an exam

> build-eval writes an exam for your AI app with you; hillclimb revises against it one change at a time, with held-out cases to catch cramming.

By Minh-Toan Ta · 3 October 2026 · 5 min read

Think back to a good exam. It looked like the job it prepared you for. The best student got the best mark, yet nobody got everything right. The same paper, marked twice, got the same grade. And those who learned past papers by heart aced the mock, then stumbled on the real thing.

An evaluation for an AI app works the same way, except you must write the exam yourself, mark it, and revise against it without fooling yourself. Anthropic added guidance for both steps to the claude-api skill that ships with Claude Code. This article expands sheet no. 3 of my *Claude · How-to* series on consonance.fyi, with detail from the original claude.dev post.

**In one sentence:** `build-eval` builds an evaluation inside your codebase, with your approval, and `hillclimb` improves your app against it, one change at a time, while keeping some examples aside to catch overfitting.

## What makes a good exam

The claude.dev post lists four traits of a well-designed eval. Each maps onto a school memory:

- **Tasks mirror production.** An exam that looks like real life: pick tasks you care about, not those that are easy to generate or grade.
- **Stronger models score higher.** The best student gets the best mark. If a more capable model or a higher effort level doesn't do better, suspect ambiguous tasks or a miscalibrated grader.
- **Headroom at the top.** Nobody scores 100%. The strongest model at the highest effort should land well below perfect, or you can't tell whether a change helped. But the gap must not come from impossible tasks; a task that fails on every run is a giveaway.
- **Low run-to-run variance.** Same paper, same mark. Noise comes from vague tasks, a grader that changes its verdict on identical output, or leftover state (a file, a git history) that hands the agent the answer.

A subtler trap is *adversarial sampling*: a teacher who writes trick questions for a single student. Keep only the cases today's model fails, and the eval measures that model's weak spots, not what is truly hard for your app. Choose hard cases because a human judged them hard, and be able to say *why*. Real failures from bug reports and tickets are welcome, but raw user traffic can skew easy: people mostly try what they expect to work.

## build-eval: writing the exam together

Run `/claude-api build-eval` and Claude interviews you, builds the eval in your codebase, and stops for your approval at set points.

**The questions.** Claude draws inputs in this order: production transcripts (after asking about retention and sensitive data), bug reports and tickets, five to ten cases you write by hand, then cases synthesized from your codebase. A simple page lists every input and waits for your confirmation.

**The grader.** Claude proposes the cheapest grader that fits. Constrained output gets a programmatic check: exact match, a label from a fixed set, JSON matching a schema, passing tests. Open-ended output gets an LLM-as-judge, reading a rubric written as checkable claims rather than a 1-to-5 scale; the judge should not be the model under test. Claude grades a handful of cases and asks whether you would have scored any differently.

**The diagnostics.** During the baseline, the grader runs twice on the same output to see if the verdict changes, and Claude checks for timeouts, API errors and cut-off answers. If the baseline already scores about 95% or higher, it warns that the climb should target cost or latency instead.

You get back the cases, the grader, the runner, a transcript per case, and a plain results page.

## hillclimb: revising one thing at a time

With a reliable exam, `/claude-api hillclimb` revises like a disciplined student, one thing at a time. You choose what it may touch: system prompt, skills, tool descriptions, model, effort and other API parameters, or harness code. It asks for your goal (performance, or cost while performance holds), splits the cases at random into **train** and **test**, and checks that the eval's noise is smaller than the smallest gain you'd act on. Then every round runs the same loop:

_The loop: build-eval sets up the cases and the grader. hillclimb learns only from the train split; the held-out test split is scored but never read._

Claude reads the previous round's **train** transcripts and proposes one patch, fixing the root cause (rewriting a section, adding a missing rule) rather than rewording a line, so the effect can show above the noise. If train and test both rise, the patch stays. A regression is reverted. If train improves while test stays flat, Claude suspects overfitting and reverts.

When the score stalls for two or three rounds, Claude sorts each remaining failure by cause, which often uncovers an ambiguous case or a harness error. At the end, your code sits at the version that did best on the test set, with confidence intervals against the baseline. If the gain is within noise, Claude recommends against merging.

> **Cramming for the exam**
>
> Learning past papers by heart is overfitting: better on the eval than in production. An eval can leak into your harness, for example when a benchmark that rewards OCR leads the climber to add an OCR tool your real users never need. The warning sign is a train set that improves while the test set stays flat, and Claude reverts the change. Two more rules: never paste failure content into the prompt, and keep the answers structurally out of the model's reach.

## What it gave in practice

The claude.dev post shares two runs. The first targeted cost on a customer support benchmark of 44 tickets, 30 searched and 14 held out. The baseline, Opus 4.8 at high effort, scored 74.4% at 4.6 cents per ticket. After a prompt audit, Opus 5.5 at low effort reached 87.8% at 1.9 cents; Sonnet 5 at low effort, 88.9% at about 1 cent; a better prompt then took it to 98.9%. On the 14 held-out tickets, the final setup scored 90.5% against the original 78.6%, at about one fifth of the cost.

The second targeted performance on the claude-api skill itself: from 66% to about 88%, partly by adding missing features, partly by fixing flawed tasks and graders.

> **Exam already too easy?**
>
> If your baseline scores about 95% or higher, there is little left to win on quality. Aim the climb at cost or latency while performance holds. That is exactly the goal of the first run above.

## Try it

1. Update Claude Code. The claude-api skill ships inside it, so updating gets you the latest version of these commands.

   ```
   claude update
   ```

2. Build the eval inside your codebase. Claude interviews you and waits for your approval of the examples and the grader.

   ```
   /claude-api build-eval
   ```

3. Improve your app against it, towards your goal (performance, or cost while performance holds).

   ```
   /claude-api hillclimb
   ```

## Conclusion

The exam analogy is the checklist. Write a paper that looks like real life, keep the grader fair and consistent, revise one thing at a time, and always keep a mock exam you've never seen. `build-eval` and `hillclimb` don't replace your judgment: they pause where it matters, and the held-out test set keeps everyone honest, the model included.

## Sources

- [consonance.fyi · Evaluating and improving an AI app, explained with an exam](https://consonance.fyi/en/mode-d-emploi/automating-eval-design-and-hillclimbing)
- [claude.dev · Automating eval design and hillclimbing with Claude](https://claude.dev/blog/automating-eval-design-and-hillclimbing/)
- [GitHub · anthropics/skills · claude-api skill](https://github.com/anthropics/skills/tree/main/skills/claude-api)

---

- HTML: https://me.consonance.fyi/en/blog/automating-eval-design-and-hillclimbing/
- [Français](https://me.consonance.fyi/blog/automating-eval-design-and-hillclimbing.md) · [Tiếng Việt](https://me.consonance.fyi/vi/blog/automating-eval-design-and-hillclimbing.md)
- MCP: https://mcp.consonance.fyi
- Contact: tmtoan2601@gmail.com · https://www.linkedin.com/in/minhtoanta/