Evals before features

An AI workflow that works in a demo and an AI workflow you can trust with a customer are different things. The difference is evals.

What an eval is

An eval is a test that runs the workflow on a real example and checks the result. Not “does it look right to me today”, but a repeatable check you can run every time something changes.

What goes in the first set

• The normal cases. Ten or twenty real examples of the work, with the answer a person would give.
• The awkward cases. The badly formatted email, the missing field, the customer who writes in two languages.
• The cases that should stop. Inputs where the right answer is “hand this to a person”.

Why before features

If the evals come last, they get written to pass. If they come first, they decide what the workflow has to do. Every feature after that has to keep them green.

It’s slower in week two and much faster by week five.