🚨 Risks & Common Mistakes

Risks & Common Mistakes: Trusting Claude's confident-sounding output without verification is like trusting a very well-read intern who has never actually worked at your company b

Trusting Claude's confident-sounding output without verification is like trusting a very well-read intern who has never actually worked at your company but speaks with total confidence — the mechanism is exact: the intern has read every textbook (an enormous training corpus) but has zero memory of YOUR specific codebase's quirks, so a confidently delivered answer and a CORRECT answer are two independent properties that happen to often coincide, not the same thing. Here is the question worth sitting with: if Claude is right most of the time, why build a whole tab of "risks" around the rare times it's wrong? Because the failure mode isn't random noise you can average away — it's SYSTEMATICALLY invisible: a hallucinated method looks exactly like a real one, a tautological assertion looks exactly like a real check, right up until it silently fails in production. Java comparison: it's the difference between a compiler error (loud, blocks the build) and a logic bug that compiles fine and passes a rushed review (silent, ships) — most of these risks are the second kind, precisely because they're syntactically perfect. The QA stake: this whole page has taught active verification habits (run it, reproduce it, check it against the acceptance criteria) — this tab collects the concrete failure shapes those habits exist to catch, so treat it as the reference checklist you hold yourself to, not a warning meant to scare you off using the tool.

Four Risk Categories, One Discipline: Verify

Hallucination: Claude can state a nonexistent API or method with full confidence because it predicts plausible text, not verified fact — the counter, covered in the UI Automation tab, is to always verify against the compiler, IDE or official docs before running anything. Privacy: pasting customer data, tokens or credentials into any AI conversation is an exposure the moment it's sent — the counter, covered in depth in the Bug Analysis tab, is to sanitize before pasting, every time.

Copyright & company policy: some organizations restrict what code or data can be pasted into third-party AI tools, or require review before AI-assisted code ships — the counter is knowing and following your company's actual AI usage policy; "I didn't know" is not a defense a reviewer accepts. Over-reliance (skill atrophy): a tester who always asks Claude first and never builds independent debugging instincts loses the skill of spotting a wrong assertion or a bad locator on their own — the counter is using Claude to accelerate work you already understand, not to substitute for understanding you haven't built yet; the verification habits from earlier tabs ARE that understanding, in practice.

An LLM predicts plausible-looking API names; a method that matches Selenium's naming convention can be entirely invented (a hallucination), and nothing about its confident phrasing signals that.

Claude generated a positional XPath from a frozen HTML snapshot without being told to prefer a stable attribute (see the UI Automation tab) — position is never a contract a frontend team promises to preserve.

A generated assertion checked a value against itself or a trivially-true condition instead of the actual expected value — a tautology that passes regardless of the real behavior.

The raw log was pasted without scrubbing sensitive fields first — the same mistake covered in the Bug Analysis tab, shown here as a real incident shape.

LLM-generated ID-shaped numbers can, by chance, collide with a real valid identifier, especially at volume — a "make it clearly fake" instruction alone does not guarantee non-collision.

The model's training data spans years of documentation across many library versions; without being told your exact installed version, it defaults to the most commonly-seen (often older) API shape.

Very long conversations can push earlier context out of effective focus, especially across many topic switches — earlier decisions compete with everything said since, without a hard memory failure being visible.

Treating AI-generated code as exempt from the team's normal review process because "an AI already checked it" — but Claude's own output was never independently verified, violating the CI/CD tab's team-rule discipline.

Step by Step: Verifying Before Trusting

Check for hallucination