What is an eval for an AI product? It is a fixed set of real inputs, each paired with a plain rule for what a good answer looks like, that you run every time the product changes. The result is one number, the share of cases that pass, plus a list of the cases that failed. A founder who has an eval knows whether a new prompt, a new model or a new feature made the product better or worse. A founder without one finds out from customers.
Evals matter more for AI products than for ordinary software because a language model does not give the same answer twice, and a change that fixes one reply can quietly break three others. The course list on AI Entrepreneur puts evals in its last part, on growing an AI company, next to agents, moats and the economics of the business, under a course titled "Evals: Shipping AI That Works".
Why an AI product needs its own kind of check
Ordinary code is predictable. Give a function the same input and it returns the same output, so a developer can write a check once and trust it. A product built on a language model behaves differently. The same question can come back in new words each time, and a small edit to the prompt can change the tone, the length or the facts of every answer.
Researchers who study these systems say the same thing about the whole field. The authors of a 2023 paper on using strong models as judges of chat assistants write that evaluating such assistants is hard because their abilities are broad and existing benchmarks do a poor job of measuring what people actually prefer. A public benchmark tells you how a model does on someone else's questions. An eval tells you how your product does on your customers' questions.
What is an eval for an AI product made of?
Every useful eval has the same three parts.
- Cases. Real inputs your product will see: customer emails, documents, questions, form entries. Start with 20 to 50. Include the common requests and the awkward ones that have gone wrong before.
- A pass rule for each case. One sentence that says what a good answer must do or must never do. "Gives the refund window as 30 days." "Does not invent an order number." "Replies in Spanish when the customer writes in Spanish."
- A way to grade. Some rules can be checked by code, such as whether the reply contains a date or stays under 100 words. Others need a person, or a second model asked to judge the answer against the rule.
Put together, the eval is a small file of cases and rules, and a short script or spreadsheet that runs the product on each case and records pass or fail.
A worked example: a support reply assistant
A small online shop uses a language model to draft replies to customer emails. The owner wants to change the prompt so replies sound warmer. Here is how an eval keeps that change honest.
Step 1: collect cases. The owner exports 30 real emails from the last two months. Ten ask where an order is. Eight ask for a refund. Five complain about a damaged item. Four ask something the shop cannot answer, such as a question about a product it does not sell. Three are written in Spanish.
Step 2: write a pass rule for each. Refund emails pass only if the reply states the shop's 30 day refund window. Order emails pass only if the reply asks for the order number instead of making one up. The four off-subject emails pass only if the reply says the shop cannot help with that question. The Spanish emails pass only if the reply is in Spanish.
Step 3: run the current version. The owner runs all 30 emails through the current prompt and marks each reply. Say 26 of 30 pass. That 87 percent is the baseline, and the four failures are written down by name.
Step 4: change one thing and run again. The owner swaps in the warmer prompt and runs the same 30 emails. Say 25 pass. The replies do sound friendlier, but two refund replies now leave out the 30 day window, while one old failure is now fixed. Without the eval, the owner would have seen friendlier emails and shipped the change. With it, the owner sees the exact two cases that broke, adds one line to the prompt about always stating the refund window, and runs the 30 emails a third time before anything goes to customers.
The whole loop takes under an hour once the cases exist, and every future change reuses them.
Grading by rule, by hand and by a model
The cheapest grading is a rule in code. If a reply must contain "30 days", a script can look for it. Rules like these are fast, free to run and never change their minds, so use them wherever a pass can be stated that precisely.
Grading by hand is the most trustworthy for questions of tone, helpfulness or judgment. It is also slow, so it suits a small set of cases or a final check before a large release.
Between the two sits a model acting as the judge: a second language model is given the case, the reply and the pass rule, and asked whether the reply passes. The 2023 paper on LLM-as-a-judge found that strong judge models agreed with human preferences more than 80 percent of the time, about the same rate at which people agreed with each other. The same paper names the judge's weak spots: it favours answers by their position in a list, it favours longer answers, it can favour answers from its own model, and its reasoning is limited. A founder who uses a model judge should grade a sample by hand too and check that the two agree.
Measuring more than right or wrong
A reply can be correct and still be a problem. It can be rude, far too long, slow to arrive or costly to produce. A research effort called Holistic Evaluation of Language Models (HELM) makes this point at scale: it measures seven qualities, namely accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, so that the measures beyond accuracy are not dropped and the trade-offs between them are clear. The same paper reports that before HELM, models had on average been checked on only 17.9 percent of its core scenarios, and some well-known models shared not a single one.
For a small product, that lesson shrinks to a few extra columns in the eval. Next to pass or fail, record how long each reply took, roughly what it cost, and whether it broke a hard rule such as sharing private data. A change that lifts the pass rate but doubles the cost per reply is a choice to make on purpose, not something to discover on the next invoice.
Keeping an eval useful as the product grows
An eval that never changes slowly stops describing the product. Three habits keep it honest.
- Add every real failure. When a customer reports a bad answer, turn that input into a new case with a pass rule. The same mistake then cannot return unnoticed.
- Keep a few cases fixed. A small core set that never changes lets you compare this month's pass rate with last month's.
- Run it before every release. A new prompt, a new model version or a new tool connection all count as changes, even when they look small.
Evals also help when choosing between models. Run the same cases through two models and compare pass rates, cost and speed on your own work rather than on a public leaderboard.
Frequently asked questions
How many cases does a first eval need?
Twenty to fifty real cases is enough to catch most regressions in a small product. Grow the set as real failures come in, rather than writing hundreds of invented cases at the start.
Is an eval the same as a benchmark?
No. A benchmark is a shared public set of questions used to compare models in general. An eval is built from your own inputs and your own rules, so it measures how your product does at its actual job.
Can a language model grade its own answers?
It can help, but research on model judges reports biases toward longer answers, answers in certain positions and answers from the same model. Grade a sample by hand and check that the model judge agrees before trusting it alone.
When should a founder start building evals?
As soon as the product has real users and a prompt that changes. The first time a change fixes one answer and breaks another, the cases from that week become the start of the eval.
What an eval changes about building with AI
The main thing an eval changes is how decisions get made. Without one, a team argues about whether the new prompt feels better. With one, the team looks at the same 30 or 300 cases, sees which passed and which failed, and decides from evidence. The cases become a written record of what the product promises to do, built from the questions real people asked. Each new failure that enters the set is a promise the product will keep from then on, and the pass rate becomes the plainest measure of whether an AI product is getting more reliable over time.