INSTANT-ON AI QA FOR ENGINEERING TEAMSBring your QA up to AI speed.Instant-on coverageTESTERS.AI · FOR TESTERSTest at the speed of AIAI-first testingJANK.AI · FOR DEVELOPERSCatch what AI coding breaksValidate every change

A BOOK BY JASON ARBON

Test at the speed of AI.Software is easy to generate.
Confidence is hard.

How AI Tests Software is a field guide to using AI to inspect, explore, test, challenge, and qualify the software that people and AI agents build. For a deep dive into testing AI itself—including AI-based functionality and systems—read my companion book, Testing AI. Explore the TestBucks practice website and companion materials as you read.

Read the full book — free
AI CHECKS•AI TESTS•AI CONFIDENCE

THE QUALITY PROBLEM HAS BECOME THE CONTROL POINT OF SOFTWARE.

01 / THE THESIS

AI is subsuming
the old testing stack.

Test frameworks, recorded scripts, test-case databases, issue trackers, and reporting dashboards were abstractions built so people could manage software complexity. AI can work directly from intent, source, screenshots, logs, traces, and the current product state.

The reusable asset is no longer a fragile click sequence. It is the promise, the evidence contract, the authority boundary, and the decision the result can change.

01

AI moves testing from brittle scripts to durable intent and evidence.

02

The coding agent can orchestrate validation, but independent validation has to qualify the result.

03

AI will spend more compute validating, comparing, repairing, and monitoring software than initially generating it.

02 / A BETTER VOCABULARY

Four terms for a new quality system.

Stop using “AI testing” as one vague bucket. Name the capability, the interaction, the workflow, and the decision.

01

AI Checks

Use screenshots, logs, network activity, DOM, traces, and source to inspect quality evidence without touching product state.

02

AI Tests

Drive browsers, devices, APIs, data, and the whole system to prove a user or business intent.

03

AI Test Harnesses

Run the long-lived workflow: context, planning, execution, challenge, reporting, and AI Confidence.

04

AI Confidence

Decide whether the evidence is enough to ship, canary, hold, repair, or gather more proof.

03 / THE LOOP

Generate. Validate.
Learn. Repeat.

An AI Test Harness turns a fresh repository, a change, a URL, or an existing suite into a stateful quality workflow. It can discover risk, generate candidates, execute them through any adapter, challenge its own findings, and continue until AI Confidence converges.

LONG-RUNNING QA WORKFLOW

  1. 01Context + riskWhat changed? What matters?
  2. 02ExecuteRun the right checks and tests.
  3. 03ChallengeTry to disprove the findings.
RESULTAI ConfidenceShip, hold, or gather more evidence.
04 / INDEPENDENCE

The creator can ask
for the test. It cannot
be the only judge.

No one buys a house without an independent inspector. No one accepts a bridge because the contractor says the concrete looks good. AI-generated software needs the same separation.

The coding agent orchestrates from the richest possible context. An independent cloud validation plane gathers its own evidence, challenges the result, and qualifies what is ready to ship.

Independent evidence is not bureaucracy. It is the product.

DRAFT READERS & CONTRIBUTORS

A better book, thanks to you.

Thank you to the readers who volunteered to help shape this draft and agreed to be recognized here. Your time, questions, and feedback matter to me.

A star marks feedback incorporated into the book so far.

Listed with permission. Recognition is a thank-you, not an endorsement. This list will grow as I confirm permissions and incorporate more feedback.

START HERE

Test at the speed of AI.
Qualify what ships.

Join the early-access list for book updates and new material.