🛠️ Build Your Own Test Agent

Build Your Own Test Agent: Building this small agent is like assembling a LEGO set out of pieces you already own separately: the messages list from the OpenAI API tab, the tool J

Building this small agent is like assembling a LEGO set out of pieces you already own separately: the messages list from the OpenAI API tab, the tool JSON schema from Function Calling, and the perceive-decide-act-observe loop from What Is an Agent — building an agent from scratch is not learning a NEW concept, it is wiring three concepts you already understand into a working while loop. Here is the question worth sitting with: if none of the individual pieces (an API call, a JSON tool schema, a while loop) are new, why does the RESULT feel like a completely different, more impressive thing than any one piece alone? Because an agent's power isn't in any single component, it's in the LOOP that lets the model see real, unpredictable results — the actual log content, the actual function output — and adjust its next decision accordingly; that feedback loop is qualitatively different from a single request/response call, even though it's built from nothing but repeated single calls. Java comparison: this is like building a simple state machine or a small interpreter out of nothing but a switch statement and a while loop — no single line is advanced, but the assembled whole exhibits behavior (looping, branching based on real input) that a single method call never could. The QA stake: this small, real, roughly 50-line script IS the mechanism behind every "AI test agent" a vendor might sell you — once you have built the toy version yourself, you can evaluate a commercial one by asking exactly what its loop, tools and safety limits actually are, instead of trusting the marketing.

The Task: A Flaky Test Report Agent

The agent reads a test log file already on disk, decides whether it describes a flaky test (an intermittent failure pattern), and if so, calls a report_flaky_test tool — the same JSON schema from the Function Calling tab — with the test name and a reason; the tool is a REAL Python function that appends a line to a report file, not a simulation. The agent's ONLY permission is: read the log file, call this one specific reporting tool. It cannot delete anything, modify the test itself, or call any other function — the narrowest permission the task needs, the exact discipline covered on the Claude AI page.

Piece 1: Setup and Reading the Log

Why Does with open(...) as f: Close the File Automatically?

The OpenAI() call here doesn't send…

The OpenAI() call here doesn't SEND any API request YET — it just PREPARES the client OBJECT, READS the API key from the environment variable, and WAITS.

The with open(...) as f: block…

The with open(...) as f: block uses Python's "context manager" MECHANISM — the file gets CLOSED automatically once the code INSIDE the block finishes (even if an ERROR is thrown), you never NEED to call f.close() by hand.

The "r" mode opens the file…

The "r" mode opens the file for READING ONLY — attempting to WRITE THROWS an error, which is a safety layer that PREVENTS the agent from accidentally MODIFYING this file.

Without EXPLICITLY setting encoding="utf-8"…

Without EXPLICITLY setting encoding="utf-8", a DIFFERENT default encoding can be USED depending on the operating system — this can SILENTLY corrupt reading if the log file contains non-ASCII characters.

f.read() loads the ENTIRE file…