For product managers shipping AI features

Evals for PMs. Know when your AI feature is ready to ship.

PMs already have the judgment. EvalSpree turns it into a repeatable test.

  1. 1

    Teach by example

    Criteria come from answers you have already judged.

    Mark a few outputs good or bad and say why in one line. EvalSpree drafts the criteria from those reasons, so you never start from a blank rubric. You can add, edit, or split a criterion while you are still labeling. When a criterion changes, earlier labels that may need another look are flagged.

  2. 2

    Trust the judge

    The judge has to catch bad answers, not only good ones.

    You and the judge score the same sample. You see how often the judge passes good answers, and how often it catches bad ones. It is trusted only when both are at least 80%. A judge that always says "pass" can look 90% accurate while missing every bad answer.

  3. 3

    Make the launch call

    The report ends with your name.

    Each criterion has a pass-rate bar based on its risk: 80% for low, 90% for medium, and 98% for high. You can edit the bar. High-risk checks, such as legal, financial, or safety, always need a person. The report ends when you record Approve for launch or Not ready, with your name and the time.

Start a project

The sample opens a private copy of a made-up onboarding assistant with about 30 answers, including a few that should not ship. EvalSpree is free. You can score a human run without an AI key. A judge run uses your key, and your provider bills those calls.

Questions

What are AI evals?

An eval is a repeatable check of an AI feature. You collect answers the feature produced, mark some of them good or bad, and turn those judgments into criteria the feature can be scored against before you ship.

Why do product managers need them?

Product managers already know which answers are acceptable. EvalSpree turns that judgment into a test you can run again when the prompt, the model, or the feature changes, and it records the launch decision under your name.

How is the judge checked?

You and the judge score the same sample. A criterion is trusted only when the judge passes good answers at least 80% of the time and catches bad answers at least 80% of the time. Until that criterion has at least three answers you marked as failing, the judge stays unverified, its results go to the review queue, and Approve for launch stays off unless a person has reviewed every result for that criterion.

Is EvalSpree free?

Yes. You can use EvalSpree without any AI key: you and your reviewers score the answers, and rule checks run automatically. If you want an AI judge to score at scale, add your own Anthropic or OpenAI key, and that provider bills the judge calls.

What happens to my data?

Test cases, criteria, and reasons stay in your project. Analytics records events such as a page view or a finished run, and does not receive case text, criteria, reasons, API keys, names, or email addresses. API keys are encrypted. Newsletter updates are sent only after you confirm, and every email has an unsubscribe link.