300 AI-Generated Tests May Provide Less Assurance Than 40 Well-Designed Tests

Generative AI can create test cases faster than a human tester. It can create a lot of them. But a large test suite is not automatically a strong test suite.

A recent paper in the ITEA Journal looks at a problem software teams are going to face more often: when AI generates the tests, how do you know the tests themselves are good enough to trust?

The answer is not to stop using AI for testing. It is to treat AI-generated tests as proposed test artifacts that still have to earn their place as evidence.

Andrew Pollner, president of ASTQB, published Human Oversight for AI-Generated Test Artifacts in the June 2026 ITEA Journal. It synthesizes developer survey data, research on LLM-generated unit tests, NIST guidance, and software testing practices to identify recurring weaknesses in AI-generated test artifacts, then proposes a six-stage process for reviewing them.

The problem is not test generation. It is test evidence.

Generating a test case and demonstrating that something has been adequately tested are two different things.

Does it challenge an important requirement? Does it exercise a meaningful risk? Would it actually detect a defect? Does it cover conditions where the software is likely to fail?

Pollner makes an important distinction between artifact volume and evidentiary strength. The ITEA paper gives the example of 300 automatically generated tests appearing more convincing than 40 human-designed tests even when the smaller suite does a better job covering operational risks, interfaces, edge conditions, and failure modes.

That is an example, not the result of an experiment comparing exactly 300 AI tests with 40 human tests. But it illustrates a real problem.

It is much easier to count tests than to measure whether they are good tests.

AI-generated tests can look better than they are

The paper identifies several recurring weaknesses that can make an AI-generated test suite look comprehensive while important risks remain untested.

1. Too much happy-path testing

Generative AI tends toward nominal scenarios: a login with the correct password, an order that completes with valid information, a request that receives the expected response. Those cases matter, but many serious defects live outside that path.

What happens when data is invalid? A dependency stops responding? A process is interrupted halfway through? A user performs actions in an unexpected order?

Negative conditions, exceptional behavior, and complex state sequences get less attention, and a test suite can grow very large without becoming proportionally better at finding failures.

2. Boundary values and partitions can be missed

Good testers deliberately look for the edges.

If an input accepts values from 1 through 100, testing 25, 50, and 75 is not enough. Values such as 0, 1, 100, 101, invalid data types, empty values, and meaningful partitions may be much more revealing.

The paper points to research showing that LLMs can struggle to infer the inputs needed to exercise all conditions in complex methods unless coverage goals are made more explicit.

AI can generate many valid-looking inputs without selecting the inputs most likely to expose a defect.

3. State transitions are easy to overlook

A lot of real software is not just input followed by output. It has state.

Think about authentication, payments, reservations, account recovery, workflow approvals, safety controls, or any process where what happened one step earlier changes what should happen next.

The paper specifically calls out transitions such as retries after timeouts, partial rollbacks, interrupted authorization flows, and recovery from degraded states as conditions that can be underrepresented in generated test suites.

Research cited by Pollner on project-level LLM test generation found that even frontier models produced fundamental errors, including compilation failures and cascading test breakdowns. The underlying ProjectTest benchmark evaluated nine frontier LLMs across Python, Java, and JavaScript projects.

4. A passing test can still be a bad test

This is the most important point in the paper.

A test can execute successfully and still provide weak evidence. The assertion may be wrong, the expected result may be incomplete, or the test may check that an API returned a 200 response without verifying whether the correct authorization state changed behind it.

Pollner cites Bhatia et al. (2023), a study comparing ChatGPT-generated unit tests with the automated tool Pynguin. That study found comparable coverage in some cases, but about one-third of the assertions generated by ChatGPT were incorrect for some categories.

Code coverage alone cannot answer whether a test would actually detect the failure you care about. The ITEA paper calls this oracle and assertion validation: a trustworthy test needs meaningful pass/fail logic, not just executable code.

5. Nonfunctional risks may disappear

A generated suite can provide extensive functional coverage while saying very little about the risks that matter most in production. This comes up in auditing AI-generated code too: the functional surface looks clean while security, performance, and integration gaps sit underneath.

In testing, that blind spot shows up across:

• Security
• Performance
• Resilience
• Accessibility
• Interoperability
• Scalability
• Auditability

Pollner argues that reviewers should determine whether these areas were deliberately excluded based on risk or simply never considered. There is a meaningful difference between a team that decided security testing was out of scope for a particular release and a team where nobody asked the AI to think about it.

Six checks before trusting AI-generated tests

The ITEA paper proposes a six-stage oversight framework built on a simple premise: AI-generated test artifacts begin as candidate output and become trusted test evidence only after appropriate review.

A useful example: more than 100 authentication tests

The paper demonstrates its framework using an example of an AI-generated test suite for a financial-services authentication platform.

On first inspection, the suite looks good. It contains more than 100 tests, high execution success, broad line coverage, login tests, password-reset tests, lockout checks, and multifactor authentication tests.

Then reviewers look deeper and find missing traceability to fraud-detection workflows, session timeouts, audit logging, and partially completed authentication flows.

They find missing boundary and state conditions such as maximum-length credentials, malformed tokens, simultaneous login attempts, degraded networks, interrupted multifactor authentication, and retry behavior.

They also find assertions that check response codes without confirming backend authorization state, session invalidation, audit events, or privilege assignments.

Finally, the suite provides little evidence of performance stress testing, accessibility, resilience under degraded dependencies, or adversarial testing. Nothing about this example says AI failed to produce useful tests. The problem is that the initial appearance of completeness was stronger than the actual evidence.

Better metrics for AI-assisted testing

AI also makes some traditional testing metrics easier to mislead without trying. If a tool can generate another 100 tests in minutes, test count becomes less meaningful as a signal of quality.

Pollner suggests looking beyond test volume, line coverage, pass rates, and generation speed toward measures such as:

• Defect-detection effectiveness
• Coverage of high-risk requirements with reviewed traceability
• Escaped defects
• Mutation resistance where appropriate
• How often AI-generated tests require substantial human rework
• Review time required for an artifact before it is accepted

The question changes from "How much testing did AI generate?" to "Did AI improve the quality of the evidence?" That is a harder metric, but it is the one that matters.

AI should accelerate test design, not replace test judgment

None of this is an argument against using generative AI in testing. Pollner explicitly argues against both extremes: banning AI-generated testing and treating AI as a replacement for test-design expertise.

AI can help draft test matrices, propose negative cases, generate test harness code, expand a suite, or update tests when requirements change.

The mistake is assuming that because AI made the artifact quickly, the artifact has already been validated. Generation and assurance are separate steps.

That distinction extends to who does the reviewing. Developers, test engineers, and managers are not interchangeable in an AI-assisted workflow. Developers are well suited to using AI for rapid scaffolding and local checks. Independent testers bring the analytical distance to challenge assumptions, spot missing partitions, and assess operational realism. Test managers own the review gates, escalation criteria, and evidence standards.

Where AI Assurance Pro fits

Software testers now need two things that used to be more separate: classical test-design skills like boundary value analysis, equivalence partitioning, state-transition testing, and nonfunctional test design, and a working understanding of how generative AI changes the work, including how to use AI-generated output without automatically trusting it.

Pollner's practical model makes exactly that point: effective review requires both, not one or the other.

ASTQB AI Assurance Pro™ brings those areas together through three ISTQB certifications: ISTQB Foundation Level, ISTQB AI Testing, and ISTQB Testing with Generative AI. The credential does not prove that every test someone reviews will be correct. What matters is whether the person doing the reviewing understands how to challenge what AI produced. If you are a tester figuring out where this fits in your day-to-day work, the AI Assurance Pro for testers page is a good place to start.

Before accepting AI-generated tests, ask

• Which model or tool generated it?
• What requirements, prompts, code, and other context did the AI receive?
• Which requirement, interface, risk, or failure mode does the test address?
• Are negative cases and boundary conditions represented?
• Are important state transitions covered?
• Would the assertions actually detect a meaningful failure?
• Have relevant nonfunctional risks been considered?
• What assumptions did the AI make?
• Has a qualified human reviewed the artifact?
• Is the level of review appropriate for the consequences of getting it wrong?

AI can make test creation dramatically faster, but the part that does not go away is deciding whether those tests deserve to be trusted.