📉 Multi-turn Conversation & Drift Testing

Multi-turn Conversation & Drift Testing: Testing a chatbot across a long conversation is reviewing a full customer-service call recording, not grading a single transcript line —

Testing a chatbot across a long conversation is reviewing a full customer-service call recording, not grading a single transcript line — and the mechanism matches one-to-one, not loosely: a QA auditor listening to a 45-minute call does not just check "was the first answer correct?", they listen for whether tone, policy adherence and topic focus hold steady from minute one to minute forty-five, because social pressure and rapport build up over the CALL, not within any single sentence. Here is the question worth sitting on: if a chatbot answers turn 1 perfectly and is running the exact same system prompt on turn 20 — nothing about its rules has changed — why would its behavior on turn 20 ever be any different? Because every token the model generates is conditioned on the entire prior context, not just the system prompt in isolation; as more turns accumulate — especially emotionally loaded or persuasive ones — that accumulated context genuinely shifts the probability distribution over the model's next output, the exact attention-dilution mechanism from the previous tab, now happening across real conversation turns instead of abstractly. Java comparison: this is not one assertEquals per request — it is closer to an integration test that asserts an invariant HOLDS across a sequence of stateful operations (an account balance never goes negative across N transactions), where a single isolated-request test tells you nothing about what accumulates. The QA stake: a support bot that holds its policy perfectly on turns 1 through 3 and breaks it by turn 10 passes every single-message test suite ever written, while still costing real money in unauthorized discounts the moment a real customer reaches turn 10 — which is exactly the failure the Drift Meter below lets you watch happen, turn by turn, instead of discovering it in a production incident report.

Why Turn 1 Isn't Turn 20

A single-turn regression test starts from a fresh, empty context every time — it can never reproduce a failure that only emerges from several turns of accumulated pressure, because by construction it never accumulates anything. This is why "just add more turn-1 tests" cannot catch drift: the failure mode does not live in any individual turn, it lives in the trajectory across turns. A real multi-turn test suite scripts an escalating scenario — a customer who starts reasonable and gradually applies social or emotional pressure over several turns — and checks the SAME constraint at every turn, specifically watching for the turn where adherence crosses a threshold.

Below, reveal the conversation one turn at a time and watch three metrics update after every assistant reply: consistency, on-topic relevance, and constraint adherence. Notice the softening at turn 3 — the assistant's language shifts from a flat rule statement to a hedged "I want to help, but" — before the actual rule violation happens at turn 4. That softening is not noise; it is the same accumulating-context mechanism already visibly shifting the model's output, just not yet enough to flip the final decision. Catching that early signal, rather than only the eventual break, is what turns a drift meter into an early-warning test instead of a post-mortem.

🎬 Why Turn 1 Isn't Turn 20

Turn 1: Clear Instruction

Every Turn Gets Appended to History

Conflicting Instruction at Turn 8

Context Is Now Noisy

Turn 20: Inconsistent Answer

At turn 1, a clear instruction is given: "answer only in Turkish."

Every new turn gets appended ON TOP of ALL previous conversation — history never gets erased, it only GROWS.

At turn 8 the user accidentally gives a conflicting instruction: "actually English is fine too."

The context now contains TWO conflicting instructions + dozens of intermediate messages — the model CAN'T tell which one is still valid.