An AI agent can pass its unit tests and still take an action that the team never tested. The gap may be between a model decision and a tool call, between an approval screen and a shell command, or between one agent session and the next.
A recent empirical study looked for evidence of those gaps in 157 popular open-source agent projects. Rather than proposing a new framework and asking teams to adopt it, the researchers reviewed repositories to see what testing and safeguards were visible in practice.
The paper, A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents, is a preprint. Its results are about public repository evidence, not a complete survey of how every developer or company tests agents. For a practical companion, see the site's guide to testing LLM applications.
Most projects had tests. Coverage was the harder question.
The researchers selected 157 open-source LLM agent projects with at least 100 GitHub stars. They examined repository documentation, source code, configuration, tests, and other artifacts. In this sample, 137 projects, or 87.3%, had conventional tests or specifications.
Other visible testing was less common. The researchers classified 95 projects, or 60.5%, as having some security- or safety-oriented tests. Seventy-eight, or 49.7%, had evaluation, benchmark, adversarial, or red-team-like paths. Eight projects, or 5.1%, had identifiable repository-visible testing paths specifically for prompt injection, adversarial behavior, jailbreaks, or red teaming.
That last number needs context. The researchers could only count evidence visible in a public repository. A team might have private testing, use an external red team, or rely on deployment controls that are not represented in the code they reviewed. Eight of 157 is not evidence that 95% of AI agent projects have no adversarial testing at all.
The paper's broader finding is more useful than the smallest percentage: testing and safeguards existed, but often did not cover how the agent moved through the whole system.
An agent has more than one route to an action
Traditional software testing remains necessary. Agent projects still need tests for parsers, API clients, storage, interfaces, permissions, and ordinary regressions. But agents introduce additional paths through which a model's decision can affect the outside world.
The study organizes its review around four areas: execution surfaces, safeguards, testing artifacts, and risk scenarios. Execution surfaces include the ways an agent can be invoked or act, such as a command line, API, browser, filesystem, shell, plugin, or subagent. Safeguards include approval, permissions, sandboxing, redaction, and domain restrictions. Testing artifacts include tests, benchmarks, evaluations, and failure-oriented checks. Risk scenarios describe situations where untrusted content or agent behavior could reach sensitive actions or persistent state.
These categories help a team ask where the checks belong. A shell action might be reachable from a browser workflow, an API, a plugin, or a background process. A policy that works in one interface may not be applied to another route that reaches the same action.
The researchers observed that controls were not always applied consistently across equivalent routes. They also found that projects commonly tested features and expected workflows, while boundary, adversarial, and multi-step tool-use failures were less likely to have direct tests.
The risk chains described in the paper are candidates for review, not confirmed vulnerabilities in every project. Finding that a repository exposes a route to a sensitive action does not prove that an attacker can exploit it. It does give the project team a concrete path to inspect and test.
Test the path from decision to effect
A useful way to apply the study is to take one action with meaningful consequences and trace it through the system. For example: a user request leads the model to choose a tool, the tool receives a permission, the action changes external state, and the system records whether the change succeeded.
That chain is an assurance exercise, not a formal sequence proposed by the paper. Its purpose is to expose the seams where a test or safeguard might stop short.
Consider an agent that can send an email. A unit test may confirm that the mail tool formats an address correctly. A separate permission test may confirm that the tool asks for approval. An end-to-end test still needs to ask whether untrusted content can influence the recipient, whether a different route can call the same tool without approval, and whether the system records what was actually sent.
For each high-impact action, ask:
- Which interfaces and tools can reach it?
- Does each route apply the same permission and data-handling rules?
- Can content from a webpage, document, tool result, or previous session influence the action?
- Can the action change files, credentials, records, or other persistent state?
- Is there a negative test showing that a missing approval or scope restriction causes failure?
- Does a regression test replay the full multi-step path after a model, tool, or configuration change?
- Can the team verify the actual external result independently of the agent's report?
These checks should be grounded in the product's actual design. A plugin that only reads public reference material has a different risk profile from one that can access a signed-in browser session or execute commands.
Convert possible risk into a test case
The study's four review areas can be used together. First, inventory the surfaces where the agent receives requests and takes action. Next, record the safeguards applied at each surface. Then identify the tests that exercise those controls. Finally, describe the paths where untrusted information or a model choice could reach sensitive state.
If a path matters, make its boundary testable. State what must remain protected, what action is allowed, and what evidence will show the control worked. A test should fail if a different tool route bypasses an approval, if a permission changes after setup, or if a multi-step workflow crosses a boundary the system is supposed to enforce.
This is where end-to-end tests matter. An agent may combine retrieval, planning, tool use, files, and external APIs. Testing each component in isolation can leave the interactions unexamined. End-to-end cases can check both the expected task and the conditions under which the agent must stop, ask for approval, or report uncertainty.
The same point applies to safety controls. A repository can contain a sandbox or permission check without demonstrating that it covers all the actions that need it. Tests should connect a specific risk to the control that is meant to limit it.
What the numbers can and cannot tell us
This study describes a fixed sample of popular public projects, not all AI agents. Its repository snapshot was collected in May 2026, and projects may have changed since then. GitHub stars are a selection criterion, not a measure of security maturity. The sample also overrepresents coding and browser or computer-use agents compared with the full range of private enterprise systems.
The researchers looked at visible implementation and testing evidence. That can show where a repository appears to have a check, but it does not establish that a safeguard works at runtime. Conversely, a safeguard or test absent from the repository does not prove the maintainers lack a private process.
Read the findings as a map of patterns that merit attention, rather than a ranking of organizations. The paper's main point is about coverage: the presence of tests is a start, but teams need to know whether those tests exercise the paths where model decisions become real actions.
Testing complete agent workflows calls for traditional test design plus a working understanding of AI systems and generative AI. If that is part of your role, or a capability you are developing, ASTQB AI Assurance Pro offers a structured route through ISTQB Foundation Level, ISTQB AI Testing, and ISTQB Testing with Generative AI. These subjects can help QA professionals plan stronger checks for agent behavior and interpret the evidence those checks produce. Review the designation and its three required certifications to decide whether it supports your next step.