Your company introduces an AI agent to help manage customer support tickets. Employees can ask it to update ticket priorities, add notes, and send responses to customers.
An employee asks the agent to change a ticket's priority from medium to high.
The agent replies that the change was successful.
But did it update the correct ticket? Was the employee authorized to make the change? What if the system returned an error? And how would anyone know whether the ticket was actually updated?
These are some of the questions software testers need to answer when evaluating AI agents.
What Makes AI Agent Testing Different?
Traditional software testing checks whether an application behaves as expected. AI agent testing involves many of the same principles, but agents can introduce additional complexity.
An AI agent can choose tools, send requests to external systems, interpret responses, and decide what to do next.
For example, an agent updating a support ticket might first search for the ticket, retrieve its details, check the user's permissions, and submit the requested update.
Each step introduces opportunities for failure.
The agent might select the wrong tool, supply incorrect information, misunderstand a response, or continue when it should stop.
It could also provide an accurate-looking response even though the external system never completed the action.
The final response is only one part of the behavior that needs testing.
Research Shows Gaps in How AI Agents Are Tested
An August 2026 research paper, Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications, examined how developers test applications built around AI agents.
The researchers analyzed 2,572 test methods from 240 modules in open-source agent applications. They also interviewed 10 experienced industry practitioners.
They found that testing frequently concentrated on individual components, often using simple inputs, mocked dependencies, and limited validation. More complicated interactions and nonfunctional requirements received less attention.
The paper was accepted for the ASE 2026 conference. Its findings describe the projects and practitioners studied, not every organization developing AI agents.
The research points to a problem worth considering. Testing whether an individual API function works is different from testing whether an AI agent chooses the correct function, supplies the right information, follows the required rules, and completes the intended task.
Earlier research introduced τ-bench, a benchmark for evaluating AI agents that interact with simulated users and external tools.
Instead of judging success only by what the agent says, τ-bench compares the final database state with the expected result.
That approach is particularly useful for understanding how to test agents that perform real actions.
A Practical Example: Testing an AI Support Agent
Consider a hypothetical support-ticket system.
The AI agent can read tickets, change priorities, add internal notes, and prepare responses. Sending a response to a customer requires employee approval.
Employees can work with tickets belonging to their assigned customers, but they cannot modify another customer's records.
For testing, we create two customer accounts with separate tickets.
Customer A has ticket 1042, currently marked medium priority. Customer B has ticket 2097.
The tester asks the agent to perform several tasks and checks what happens.
| Test | Expected result | What to verify |
|---|---|---|
| Change ticket 1042 to high priority | Ticket updated | Correct ticket and stored priority |
| Update ticket 2097 without permission | Update denied | Authorization result and unchanged record |
| Change a ticket without identifying which one | Agent requests clarification if the ticket cannot be identified reliably | No unintended changes |
| Send a customer response | Agent waits for required approval | Approval record and message history |
| Add a note when the API times out | No unintended duplicate notes | Note history and retry behavior |
| Ticket contains instructions to disclose another customer's data | Unauthorized instructions are not followed | Access controls, tool calls, and outgoing data |
| Ticket update is rejected by the API | Agent reports that the update failed | API response and actual ticket status |
These are proposed tests for a fictional application, not results from a research study.
The table identifies what should happen according to the application's requirements. The next step is to execute the tests and collect evidence.
1. Test Whether the Agent Uses the Correct Tools
Start with a straightforward task.
Ask the agent to change ticket 1042 from medium to high priority.
The agent may need to retrieve the ticket before updating it. Depending on the system, it could use one tool or several.
The tester should examine which tools were called, what information was supplied, and what each tool returned.
Did the update request contain the correct ticket number? Was the priority value valid? Did the agent use a tool capable of changing the record, rather than one that only retrieves information?
Also check how the agent handles missing or ambiguous information.
If an employee says, "Change my customer's ticket to high priority," but that customer has several open tickets, the agent should not guess which record to modify.
It may need to ask a question before continuing.
Not every valid workflow will use exactly the same sequence of tools. Some agents may take different paths and still complete a task correctly.
The tester should focus on the required behavior, the permitted actions, and the final result. A particular tool sequence should be required when the application's rules make that sequence necessary.
2. Check That Permissions Are Actually Enforced
Permissions deserve separate attention because an agent may have access to tools that can read or modify sensitive information.
In our example, the employee is authorized to update Customer A's tickets, but not Customer B's.
Ask the agent to change ticket 2097, which belongs to Customer B.
The expected result is that the unauthorized update does not occur.
However, checking whether the agent refuses the request is not enough.
The tester should also verify that the underlying application enforces the restriction.
What happens if the AI attempts to call the update tool anyway?
The service responsible for changing the ticket must check authorization and reject the request.
This follows guidance in the OWASP AI Agent Security Cheat Sheet, which recommends limiting tool permissions and enforcing authorization independently of the model.
An instruction telling the AI not to modify unauthorized records is not a substitute for access control.
The same principle applies to actions requiring approval.
If the support agent is allowed to draft customer messages but must receive approval before sending them, test that a message cannot be sent without valid approval.
Check the actual message system, not just the agent's response.
3. Test What Happens When Tools Fail
External systems do not always respond as expected.
An API might reject an update, return incomplete information, become unavailable, or time out.
The agent needs to handle these situations without making incorrect assumptions about what happened.
Suppose the support agent adds an internal note to a ticket. The ticketing system saves the note, but the response to the agent times out.
The agent doesn't know whether the operation succeeded.
If it immediately repeats the request, it might add the same note twice.
This is a familiar distributed-systems problem. Amazon's engineering article, Making Retries Safe With Idempotent APIs, explains how retryable operations can create unintended effects and how API designs using request identifiers can help prevent duplicates.
For AI agent testing, create controlled failure conditions and observe how the agent responds.
If an API reports a timeout, does the system check whether the operation completed before attempting it again? Does it use an idempotency mechanism when one is available? Does it accurately describe the uncertain result to the user?
If an update is explicitly rejected, the agent should not claim that the task was completed.
These tests should also examine what happens after a partial failure in a longer workflow.
For example, the agent might successfully update a ticket but fail to record an internal note. The tester needs to know whether the system reports that partial result accurately.
4. Verify the Actual Result, Not Just the Agent's Answer
An agent can produce a convincing completion message without accomplishing the requested task.
That's why testers need evidence from the systems the agent interacts with.
Return to ticket 1042.
After the agent reports that the priority was changed, retrieve the ticket directly from the test system.
Is the priority now high? Is the ticket number correct? Were any other records changed? Does the audit history show what happened?
This approach is supported by the τ-bench research, which evaluates task outcomes by examining the resulting system state rather than relying on the agent's final response.
Anthropic's engineering guide, Demystifying Evals for AI Agents, also distinguishes between an agent's execution record and the actual outcome. This is guidance from an AI developer based on its evaluation experience, not an independent study of all agent systems.
Both types of evidence are useful.
Tool-call records help testers understand what the agent attempted. The final state of the system helps establish what actually happened.
When those checks are part of a scored agent evaluation, the scoring rules need review too. See how to test an AI agent benchmark before relying on its pass rate.
For actions involving external effects, such as sending messages or creating transactions, additional records may be necessary.
A successful API response alone may not establish that every downstream action completed.
5. Test Complete Workflows and Repeat Important Cases
Individual tool tests are useful, but they don't cover every problem that can occur when an agent performs a sequence of actions.
Consider a request to find an open support ticket, change its priority, prepare a customer response, and send that response after approval.
The tester needs to check the complete workflow.
Does the agent identify the correct ticket? Does it update the priority? Does it wait for approval? Does it send the approved message to the intended recipient?
The test should also cover what happens when approval is denied or one of the steps fails.
AI agents may not behave identically every time they receive the same request. For important scenarios, repeat the test to see whether the agent consistently satisfies the requirements.
Reset the test environment to a known state between runs so previous actions do not affect later results.
Keep useful failed scenarios as regression tests. Run them again when the model, instructions, tools, permissions, or application logic changes.
For more on gaps found in existing agent projects, see What 157 AI Agent Projects Reveal About the Tests Teams Are Missing.
Tools for Testing AI Agents
Some parts of AI agent testing can be handled with existing API testing tools, automated assertions, logs, and test data.
Other checks may benefit from tools that capture agent activity, compare execution traces, run evaluation datasets, or help analyze repeated results.
AT*SQA maintains AT*Tools, a free directory of software testing tools, including a category for AI/ML and LLM Evaluation.
The directory can help testers find tools to investigate for their own projects. AT*SQA has not independently tested the listed products or verified their performance claims, so teams should evaluate each tool against their requirements.
A specialized tool can make collecting evidence easier, but the tester still needs to determine what constitutes a successful result.
AI Assurance Is Everyone's Responsibility
Testing AI agents involves more than the QA team.
Developers need to implement reliable tools and access controls. Security specialists need to assess sensitive actions and unauthorized access. Business owners need to define what the agent is allowed to do and which actions require approval.
Testers help establish whether those requirements are met and whether the system behaves correctly under normal and failure conditions.
AT*SQA discusses this broader responsibility in Everyone Needs to Know AI Assurance. Anyone using AI or relying on its results has a role in checking its work, although organizations still need clear accountability for important decisions.
As AI agents gain the ability to change data, contact customers, and carry out business processes, those responsibilities become more important.
Software testers who want to develop their AI-related testing knowledge can also explore the ASTQB AI Assurance Pro™ designation, which combines ISTQB Foundation Level, ISTQB AI Testing, and ISTQB Testing with Generative AI certifications.
The next time an AI agent reports that it completed a task, don't stop at the confirmation message.
Check what it actually did.