AI Testing Interview Questions: What to Expect and How to Answer

The questions below are representative of what comes up. They are not guaranteed to appear word for word, but the concepts behind them will. Work through each one until you can answer it without notes.

Part 1: Testing AI Systems

How is testing an LLM-powered feature different from testing traditional software?

This usually comes first. The answer covers more than one idea, and candidates who reduce it to a single concept tend to get followed up into a corner.

The most visible difference is that LLM outputs are probabilistic. Ask the same question twice and you may get two different answers, both valid. That makes the oracle problem harder: you cannot define a single expected string and check against it for open-ended natural language output. But this does not mean exact assertions disappear entirely. Structured outputs, tool arguments, schema conformance, required fields, safety guardrail behavior, and API contracts can still be tested with deterministic checks. The probabilistic layer sits on top of a system that still has contractual behavior worth verifying.

Beyond output variability, AI systems introduce several other testing challenges that traditional software does not: quality depends heavily on training and runtime data, which makes data testing part of the job; performance is statistical across a population rather than verifiable example by example; multiple answers can be acceptable for the same input; and the system can degrade silently as the world changes around it without any code changing. Those characteristics together are what change how you approach the work, not output variability alone.

What you do instead of exact-match assertions for quality is evaluate against defined criteria: is the response grounded in the source material, does it address the question, is it safe, does it meet the accuracy bar required for the use case? Thresholds for those criteria should come from business risk, manual review of calibration examples, and the acceptable failure rate for the specific application. There are no universal numbers.

How do you test a chatbot when there is no single correct answer?

The answer to this is benchmark datasets and scoring rubrics.

A benchmark dataset is a reviewed, version-controlled collection of representative inputs paired with trusted reference information: known facts, expected properties, acceptable answer criteria, or reference answers where a single correct answer exists. Some cases have exact expected outputs. Others define what a good answer must contain, what it must not contain, and which quality dimensions matter most for that case type. Build an initial set before major changes, then expand it continuously as you discover new failure modes, edge cases, and production incidents.

You score each response against your defined dimensions: is it grounded in the source material, does it address the question, is it safe, does it meet the accuracy bar the use case requires? Thresholds for each dimension should come from manual review of real examples, business risk, and the acceptable failure rate for that application. A release gate blocks deployment when scores fall below those thresholds or when a critical failure category, such as a safety violation, appears at all.

Walking through that process in an interview, dataset, dimensions, thresholds calibrated to risk, release gate, shows you have thought about how to operationalize AI testing rather than just knowing it is hard.

Walk me through how you would test a RAG application for hallucinations.

RAG stands for retrieval-augmented generation. The system retrieves relevant documents from a knowledge base and uses them as context before generating a response. The first thing to understand is that a wrong answer in a RAG system has more than one cause, and they require different fixes.

A retrieval failure means the system did not surface the document that contained the right answer. The generator had nothing useful to work with. A faithfulness failure means the relevant document was retrieved but the generator ignored it or added claims not supported by it. A factual error means the retrieved source itself was wrong or outdated. These are three separate problems: a retrieval problem, a generation problem, and a data quality problem. Conflating them makes both debugging and test design harder.

A practical testing approach runs the pipeline in layers:

Build a test dataset of questions with known answers that exist in your knowledge base.
Run the system and collect the full response plus the documents that were retrieved for each question.
Evaluate retrieval quality separately: did the right document actually get retrieved? If not, the failure is in the retriever, not the generator.
Score faithfulness on responses where retrieval succeeded: are the claims in the response supported by the retrieved context, or did the model add something on its own?
Score answer relevancy: did the response address the question that was asked, or did it drift?
Test abstention: when no relevant document exists, does the system say it does not know, or does it confabulate an answer?

The reason to separate retrieval from generation is that a low faithfulness score on a response where retrieval failed is not evidence the generator hallucinated. It may just mean the generator had nothing to work with. Diagnosing which layer failed determines where the fix goes.

What is prompt regression testing and why does it matter?

Prompt regression testing means running your prompt templates against a benchmark dataset after any change to the system, and comparing the results against a stored baseline.

The reason it matters: LLM providers update their models without always making it obvious how your specific prompts will be affected. A prompt that produced reliable outputs on one model version can perform noticeably worse on the next. Without a regression test, that degradation is invisible until a user finds it or support tickets start rising.

The practical setup is the same as software regression testing: you have a baseline, you run the same inputs after a change, and you compare the scores. The difference is that you are comparing quality scores rather than exact outputs. If faithfulness drops after a model update, that is a regression, even if no code changed.

Versioning your prompts is part of this. If you cannot point to exactly which prompt version was running at the time quality dropped, debugging becomes guesswork.

How do you detect model drift in production?

Model drift is the gradual degradation of a production model's performance over time, even without explicit changes to the system. It has two main causes.

Data drift happens when the distribution of inputs the model receives in production diverges from what it was trained on. If your product has changed significantly since a model or knowledge base was last updated, the system starts encountering inputs it was never prepared for.

Concept drift happens when the relationship between inputs and correct outputs changes, even if the inputs themselves look familiar. A fraud detection model trained before a major shift in fraud tactics will keep running without errors while becoming progressively less useful.

Neither announces itself. The model keeps running. Outputs just get worse.

Detection requires continuous monitoring in production: tracking output quality metrics over time, not just uptime and latency. When scores start trending down, you investigate. Has the input distribution shifted? Did a model provider push an update? Did user behavior change? The monitoring tells you something is wrong. The investigation tells you what.

The point to land in an interview is that deployment is not the finish line. Quality is something you measure and maintain after launch, not something you certify once and walk away from.

What is the difference between testing and evaluation in an LLM context?

Deterministic checks verify contracts and invariants. Whether the API returned a response, whether the response came back within the latency threshold, whether a safety guardrail fired, whether a structured output conforms to its schema: these produce binary results and can be automated as conventional tests.

Empirical evaluation measures behavior across a population using metrics, rubrics, and quality scores. How grounded are responses across the benchmark dataset? How often does the system drift off topic? How does performance vary across user segments or languages? These produce numbers and distributions, not verdicts.

A complete AI quality program needs both. Deterministic checks catch hard failures. Evaluation catches quality degradation. A system can pass every automated check while its evaluation scores quietly trend downward. You need the evaluation layer to catch that before users do.

How would you design a test strategy for an AI agent?

An AI agent does not just generate text. It makes decisions, calls external tools, and takes actions that can have real consequences. That changes the testing surface significantly compared to a chatbot.

A test strategy for an agent needs to cover several things that chatbot testing does not:

Scope adherence: Does the agent stay within its intended boundaries, or does it attempt actions it was not designed to take?
Tool failure handling: When an external tool the agent calls fails or returns an error, does the agent handle it gracefully, or does it hallucinate a successful result and proceed as if nothing went wrong?
Plan correctness: Does the full multi-step plan reach the right outcome? Each step can look reasonable in isolation while the overall plan goes in the wrong direction.
Irreversible actions: Does the agent have appropriate checkpoints before taking actions that cannot be undone? Are those checkpoints tested for bypass?
Adversarial inputs: Can malicious content in the agent's environment redirect its behavior? This is the prompt injection problem applied to autonomous systems, and it is a serious risk for any agent that processes external content as part of its workflow.

The failure modes for agents are different from the failure modes for chatbots. A test strategy scoped only to output quality will miss most of them.

Part 2: Testing With AI Tools

How are you using AI in your current testing workflow?

Be specific. Name what you actually use AI tools for and describe a real workflow. Candidates who say they use AI to speed things up without specifics signal that they have not used these tools seriously enough to have opinions about them.

More importantly: include something the tool got wrong and what you did about it. The people interviewing you use these tools and know where they fail. A candidate who only talks about how useful AI is without acknowledging its failure modes comes across as either inexperienced or incurious. Neither is what a hiring manager wants on a QA team.

A useful structure: describe what you use it for, when it works well, where it falls down, and what you do to compensate. That answer demonstrates practical experience and critical judgment at the same time.

What do you do when AI-generated test cases look complete but feel wrong?

AI-generated test cases tend to cover obvious paths and miss edge cases that come from understanding business context, user behavior, or the history of a particular feature. A generated test suite can look thorough while missing the failures that actually happen in production.

The discipline is to treat generated test cases as a starting point. You review them against requirements and add cases the AI would not know to generate: cases based on past production incidents, cases for unusual user populations, cases tied to regulatory requirements specific to your domain. The generated cases save time on the obvious coverage. Your judgment covers the rest.

What you want to avoid is approving a test suite because it looks complete. Looking complete and being complete are different things, and AI tools are particularly good at making incomplete coverage look thorough.

How do you review AI-generated code for testability?

Testability is not the same as correctness. Code can work and still be difficult to test. Testability is about whether you can control the code's inputs, observe its outputs, isolate it from dependencies, and put it into a known state before each test.

When reviewing AI-generated code for testability, look for:

Dependency injection: Are external dependencies hardcoded, or can they be substituted in tests? Code that instantiates its own database connections, API clients, or file handles inside functions is difficult to isolate.
Observability: Can you verify what the code actually did, or only whether it ran without error? Side effects that produce no observable output are hard to assert against.
State control: Can tests set up a known starting state and clean up after themselves, or does the code rely on shared or global state that persists between runs?
Seams: Are there clear boundaries where test doubles can be inserted, or is the logic tightly coupled to infrastructure?
Error propagation: Does the code surface failures visibly, or does it swallow exceptions silently?

AI-generated code has specific additional failure modes worth checking: fabricated library references that do not exist, API calls using method signatures from older or incorrect versions, and generated tests that mirror the implementation rather than independently verifying behavior. A test that was generated from the same prompt as the code it tests is likely to share the same misunderstandings.

The Credential Question

What have you done to formally develop your AI testing skills?

This comes toward the end of the interview. It sounds like a soft question but it has a real answer: either you have done something structured or you have not.

Candidates who say they have been reading articles and following the space get acknowledged and moved past. Candidates who can point to structured, exam-based credentials that specifically cover AI testing are in a different position. The ASTQB AI Assurance Pro designation requires passing three ISTQB certifications: Foundation Level, which covers core testing principles and terminology; AI Testing, which covers testing AI and ML systems including data testing, model testing, bias, drift, and statistical approaches; and Testing with Generative AI, which covers prompt engineering, hallucination measurement, guardrail testing, RAG evaluation, AI risk management, and using generative AI responsibly in a test organization. Those are two distinct skill sets and the designation requires both. Holding it gives you a specific, accurate answer to a question most candidates answer vaguely.

It is also worth noting that the ISTQB exams are available through AT*SQA and can be taken online, which means the path is accessible regardless of where you are or what your schedule looks like. If you are currently building toward the designation, say so and describe where you are in the sequence. That is still a better answer than saying you keep up with the space.

What These Questions Are Really Probing

Across all of the above, interviewers are trying to answer one question: does this person understand that AI systems fail differently than traditional software, and have they built habits around catching those failures?

The specific questions vary. The underlying screen does not. Candidates who have done the deliberate work to develop these skills, and who can demonstrate that clearly in an interview, are in short supply. That gap is not closing quickly.