What stays the same
The core job does not change because the system includes an AI component. You are still trying to find out whether the software behaves the way it should, catch what can go wrong before users do, and give the team clear information about quality and risk.
Most of the code inside an AI-enabled product is not AI at all. There is routing logic, API calls, data validation, authentication, access controls, logging, error handling, and any actions the system takes in the real world like writing to a database or sending a notification. All of that behaves like standard software. It gets unit tests, integration tests, and regression coverage the same as any other system. The fact that there is a language model somewhere in the stack does not change how you test those parts.
ISTQB's CT-AI v2.0 syllabus is explicit on this point. Testing an AI system means applying traditional testing methods to the deterministic components and adding specific techniques for the AI components themselves. The ratio shifts depending on how much of the product is AI-driven, but both skill sets are always in play.
What changes when the system includes an AI component
Most software is deterministic. Same input, same output. That is what makes traditional testing work. You define the correct output before the test runs, compare the actual result against it, and get a pass or a fail.
AI systems built on machine learning often do not work that way. A large language model given the same prompt twice can return two different responses. Both might be acceptable. One might be better. Or one might be subtly wrong in a way that no string comparison will ever catch. The ISTQB CT-AI v2.0 syllabus names this directly as one of the characteristics testers need to account for: probabilistic behavior and non-determinism.
This does not make testing AI systems harder in a general sense. It means a few of the standard techniques need to be adapted or supplemented for the parts of the system where non-determinism shows up.
Defining what correct looks like. In a standard test, you specify the expected output in advance. For AI outputs, that is often not possible. There may be several acceptable responses to the same input. So instead of checking whether the output matches a specific value, you define a range of what acceptable looks like and evaluate whether the output consistently falls within it. NIST describes this challenge in its AI TEVV work: trustworthy AI requires measurement across characteristics like accuracy, reliability, robustness, safety, and bias. Each of those needs its own evaluation approach rather than a simple pass/fail check against a known value.
Testing for AI-specific failure modes. Traditional software fails in ways testers are trained to catch: wrong output, crashes, broken logic. AI systems can fail in a few additional ways that have no direct equivalent in standard software.
Hallucination is the clearest example. A model can produce information that sounds completely reasonable but is factually wrong or invented. Standard functional tests will not catch this because the system did return a response. You need specific methods to check whether outputs are grounded and accurate. What Is Hallucination Testing? covers those methods in detail.
Bias is another. A model might respond differently to similar requests depending on phrasing, name choices, or demographic signals in the input. Checking for that requires specific test design beyond what a standard regression suite covers. The OWASP AI Testing Guide, released in November 2025, frames the goal clearly: the objective is not just security. It is AI trustworthiness, which requires a broader set of checks than any security scan provides.
Testing for prompt injection. When software accepts user input and passes it to an AI model, there is a risk that a crafted input manipulates the model into doing something it should not. This is called prompt injection and it is specific to AI systems. The mindset behind it is familiar to anyone who has done security testing, even if the mechanics are different. What Is Prompt Injection? explains how it works and what to test for.
Monitoring in production. Traditional software testing is mostly a pre-release activity. You test before you ship, then monitor for bugs once the system is live. AI systems need a bit more ongoing attention because model behavior can shift over time even when the code has not changed. This is called model drift. A system that passed all its quality checks at launch may behave differently six months later because the distribution of real user inputs shifted, or because the underlying model was updated by its provider.
NIST built its ARIA program around this problem. ARIA uses a three-level structure for evaluation: model testing, red-teaming, and field testing. The reason for all three is that some things only surface once real users interact with the system in realistic settings. Pre-release testing alone cannot replicate that.
Side by side: what changes and what does not
| Software testing | With AI components added | |
|---|---|---|
| Core goal | Verify the system behaves correctly and safely | Same |
| Deterministic components | Unit tests, integration tests, regression suites | Same |
| Expected outputs | Defined in advance | Defined as an acceptable range for AI outputs |
| Failure modes | Wrong output, crash, broken logic | Also includes hallucination, bias, prompt injection |
| Adversarial inputs | Edge cases and unexpected user behavior | Also includes prompt injection attempts |
| Coverage | Code paths and branches | Also includes behavioral scenarios and input diversity |
| Production monitoring | Bug tracking and performance monitoring | Also includes drift monitoring and output sampling |
| Certification path | ISTQB Foundation Level | Foundation Level plus CT-AI and CT-GenAI |
What the extra techniques look like in practice
The ISTQB CT-AI v2.0 syllabus covers the skills that go on top of a standard testing foundation. None of them replace what a software tester already knows. They build on it.
Probabilistic evaluation. Instead of checking whether an output matches a fixed value, you assess whether it consistently falls within a defined acceptable range. This means building evaluation sets that reflect real user inputs and setting quality thresholds the system needs to meet over time.
Hallucination detection. Checking whether model outputs can be traced back to source material and flagging responses that appear to contain invented or unsupported information. How to Test LLM Applications and LLM Testing for QA Engineers cover specific methods for this.
Bias and fairness testing. Evaluating whether the model behaves consistently across different groups, phrasings, and input types. This matters most for systems that affect hiring, finance, healthcare, or anything customer-facing.
Adversarial testing. Actively trying to manipulate model behavior through edge-case inputs and prompt injection attempts. The mindset is similar to security testing: assume someone will try to break it and design tests that reflect that.
Drift monitoring. Tracking whether model behavior changes over time and building a process to catch those changes before they cause problems in production.
These are teachable skills, and they are the skills covered by the ISTQB AI Testing certification and the ISTQB Testing with Generative AI certification. Both certifications sit inside the ASTQB AI Assurance Pro path alongside the Foundation Level.
Why this matters for software testers right now
AI components are showing up in more products. Not as research projects but as everyday features: support bots, document summarizers, coding assistants, recommendation systems, content moderation tools. For QA teams, that means a growing share of what needs to be tested includes an AI layer.
The good news for experienced software testers is that most of what they know transfers directly. The techniques above are additions. A tester who understands risk, knows how to design a test suite, and thinks like a skeptical user already has most of what they need. What the AI layer adds is a specific set of failure modes to check for and a few evaluation methods that work differently from standard assertions.
How AI Is Changing Software Testing covers what QA teams are actually doing about this right now. What Is AI Testing? goes deeper on the discipline itself. For the career side, How to Become an AI Tester and the AI Tester Job Description cover what the role looks like day to day.
For managers thinking about team skills and hiring, How to Evaluate AI Testing Skills When Hiring and the SQA Talent Assessment page cover that side of the question. The broader context for what the software quality role looks like today is in What Is an SQA Expert?
Sources
- ISTQB Certified Tester AI Testing (CT-AI) v2.0 — May 2026
- ISTQB Certified Tester Testing with Generative AI (CT-GenAI) — July 2025
- NIST AI Test, Evaluation, Validation and Verification (TEVV)
- NIST Assessing Risks and Impacts of AI (ARIA) — launched May 2024
- OWASP AI Testing Guide v1 — November 2025