How to Test AI Features With Non-Deterministic Outputs

Ask the same question to the same model twice and you can get two different, both correct, answers. A test written the traditional way, one input mapped to one expected output, fails against the second answer even though nothing is actually broken.

That's not a testing problem you fix by pinning the temperature to zero. It's a sign the assertion itself is the wrong tool for this layer of the system.

Separate what's deterministic from what isn't

Not everything in an AI feature is non-deterministic, and treating all of it that way throws away your easiest wins.

The scaffolding around the model, the API contract, the tool-call schema, authentication, rate limiting, error handling when the model call fails, should get regular, exact-match tests. If a tool call comes back with a field missing or the wrong type, that's a hard failure, not a judgment call. Write those the way you'd test any other API. Our API testing practices apply here without modification.

The generative core, the actual text or decision the model produces, is where you need a different approach entirely.

What to test instead of exact output

Structural checks first. Before asking whether the content is good, ask whether the response is even shaped correctly: valid JSON if JSON was requested, the right fields present, the tool the model was supposed to call actually called. These are still deterministic and still cheap. Most bugs in production AI features are structural failures caught here, not subtle content problems.

Golden sets with acceptable ranges, not single expected strings. Build a fixed set of representative inputs and, for each one, define what a passing response has to contain or avoid, not what it has to say word for word. "Must mention the refund policy" is testable. "Must say exactly this sentence" is not.

LLM-as-judge, used carefully. A second model can score a response against a rubric (accuracy, tone, whether it followed instructions) more cheaply than a human reviewing every regression run. It's also its own source of noise: a judge model has its own failure modes and its own cost. Treat judge scores as a percentage pass threshold across the golden set, not a per-case pass/fail gate, and don't expect 100%. A model that hits 92% on a stable rubric and holds that number release over release is a suite doing its job.

Regression testing for prompts

Prompt changes need the same discipline as code changes: run the new prompt against the full golden set before it ships, and compare the pass rate to the baseline. A prompt tweak that improves one example and quietly breaks twelve others is the AI-feature equivalent of a regression, and it's invisible without a fixed set to check against.

What to do when an AI tool keeps giving incorrect outputs

Report it and document it. Don't ignore it, and don't assume it will fix itself after the next model update. A useful report, written like any good bug report, includes:

• The exact input, including the system prompt or prompt version.
• The incorrect output, copied in full.
• The model name and version, and settings like temperature.
• How often it happens when you run the same input several times.

Then add the case to the golden set. A repeated wrong answer that gets reported but never added to the suite can come back silently after the next prompt change.

A related check: ask the same question with slightly different wording and compare the facts in the answers. If rephrasing changes the facts, the feature isn't reliable on that topic, even when each answer sounds confident.

What this doesn't solve

None of this catches a model that's confidently wrong about something your golden set didn't anticipate. Eval-based testing is only as good as the scenarios you thought to include, which is exactly why exploratory testing still matters even more here than in a deterministic UI: someone has to go looking for the input nobody wrote a golden case for.