⚖️ Deterministic vs Stochastic Testing

Deterministic vs Stochastic Testing: A deterministic test is a vending machine; a stochastic test is a job interview — and the mechanism matches one-to-one, not loosely.

A deterministic test is a vending machine; a stochastic test is a job interview — and the mechanism matches one-to-one, not loosely. Press B4 on a vending machine and the same chocolate bar drops every single time; a broken machine is one that does NOT do that. Ask an interview candidate the same question on Monday and Friday and you get two differently-worded answers — yet a good interviewer still decides fairly, because they score against a rubric ("did they cover trade-offs? give a concrete example?"), not against one memorized sentence. Here is the question worth sitting on: if the answer legitimately changes every run, what does "the test passed" even mean — and how do you write an assertion for a moving target? Because the moment you stop asserting "output == expected" and start asserting "output satisfies rubric ≥ threshold", pass/fail becomes measurable again without pretending the output is fixed. Java comparison: a deterministic test is assertEquals(expected, actual) — one true value; a stochastic test is closer to a code review — there is no single correct line, you grade against a checklist and accept several valid shapes. The QA stake: mixing these two worlds is where AI test suites rot — asserting an exact chatbot string makes a green suite flaky and useless, while "eyeballing" a login redirect wastes an LLM call on something a fixed assertion nails for free. Knowing which world a check lives in is the first real AI-QA skill, and the lab below lets you feel the difference by re-running both sides yourself.

Two Assertions, Two Worlds

The split is not about the tool — it is about whether the output is fixed. A deterministic check has one correct answer, so you assert equality and any deviation is a bug: URL, status code, computed total, schema shape. A stochastic check has many acceptable answers, so you cannot assert one string; instead you define a rubric (accuracy, relevance, safety…), score the output against it — usually with an LLM-as-a-judge — and gate on a threshold like "overall ≥ 4". A mature AI test plan keeps both: deterministic "gates" for anything with a fixed truth, and probabilistic "checks" for anything a model generates. Confusing the two is the most common way an AI test suite becomes either flaky (asserting exact generated text) or blind (never checking generated quality at all).

Run the left and right panels below a few times. The Playwright side returns the identical PASS on every run because the DOM state is fixed. The chatbot side returns a different answer each run — and on one of them the model invents a "30-day / same-day refund" policy that never existed: fluent, confident, and wrong. A single-string assertion would either flake on the harmless wording changes or miss the dangerous hallucination entirely; a rubric score catches the bad run while accepting the two good ones.

Deterministic (Playwright)

toHaveURL(/dashboard) — PASS

Stochastic (LLM-judge)

Testing that a login form redirects the user to /dashboard with valid credentials.

Evaluating the quality of a customer-service chatbot's answer to "where is my order?".

Verifying a REST API returns 200 and the correct JSON schema for GET /users/1.

Checking whether a RAG assistant's summary of a company policy document contains hallucinations.

Testing that a shopping cart total shows the correct amount when 3 items are added.

🎬 assertEquals Dies, assertSemantic Is Born

Run 1: "You can return within 30 days"