Evaluating Infra AI Agents: Seven Questions to Ask
How is an AI agent’s success rate measured? What does that figure include, and what might remain outside the evaluation?
In a previous experiment, I examined LLM execution across hundreds of infrastructure tasks, thousands of virtual machines, and tens of thousands of commands across eight Linux distributions. The task success rate exceeded 92 percent. Nearly half of successful runs also left an unwanted condition behind, including exposed credentials, firewall changes that did not survive a reboot, and packages installed from unvetted sources.
I use the illusion of done to describe this gap between completing the requested task and verifying the condition of the system afterward.
When evaluating an AI agent for infrastructure work, what evidence can a vendor provide about verification, security, consistency, duration, scale, and existing environments? The following seven questions provide a structure for that discussion.
1. How does your agent fail across components?
In my research, performance tended to deteriorate on components with less public documentation. I call the degree to which an LLM can reliably work with a given component its LLM-friendliness coefficient. Does the vendor observe a similar pattern? Ask for results broken down by component, version, and task type: how does performance on popular, well-documented software compare with proprietary middleware, legacy systems, and the less-documented parts of your stack?
For each of those components, ask what qualifies as a successful result and how the agent verifies it. Does it test the resulting system state against the original requirements, or treat commands completing without errors as proof that the task is done? What verification framework supports those checks, and can an independent evaluation catch cases where the agent declares success but the requirements remain unmet?
2. How does your agent handle security?
In my research, agents left private keys in temporary directories, attempted to use hallucinated repository URLs that did not resolve, and installed packages from unvetted sources. They also made firewall changes that appeared correct during execution but disappeared after a reboot.
How does the vendor test for these outcomes? Ask how credentials are scoped, stored, used, and removed, including whether they can appear in command histories, logs, temporary files, or execution traces. What checks establish that credentials are no longer accessible after the task is complete? How are package sources, permission changes, and the persistence of security configurations verified?
More broadly, what guardrails does the agent require to operate within its authorized scope? Which are built into the product, and which must you provide? How are permitted actions, prohibited actions, and requirements for human approval defined? Are those boundaries expressed as instructions to the model, enforced through external controls, or both?
What happens when guardrails are missing, misconfigured, or unavailable? Does the agent recognize their absence and stop, request intervention, or continue? When a guardrail blocks an action, does the agent respect the boundary or seek another route to the same result?
Ask for evidence of how guardrails actually alter behavior. Has the vendor run the same tasks with and without them? What changed in the agent’s decisions, actions, and outcomes? How was compliance verified throughout execution?
How do these protections differ between VMs and containers? Which depend on the surrounding environment? On VMs, are configuration changes verified after reboot? In containers, how are secrets handled across files, logs, images, and mounted volumes, and does the intended configuration survive recreation?
3. What happens if I run the same task 1,000 times?
How consistently does the agent behave when the instructions and starting conditions remain unchanged?
Ask how many times each task has been repeated and how the outcomes are classified. Across those runs, how often did the agent succeed without unwanted side effects, succeed with side effects, fail without changing the system, or leave a partially modified environment?
Even when runs achieve the same functional result, do they produce different commands, dependencies, permission changes, or security outcomes? How are those differences measured, and which would be counted as failures?
Within that distribution, what are the most severe observed outcomes, and how frequently do they occur? How much testing supports those frequency estimates, and what uncertainty remains?
4. How does your agent handle long-running tasks?
What changes in the agent’s behavior as execution time increases?
Ask how it retains the original objective, constraints, and acceptance criteria throughout a long operation. How does it track completed work and distinguish current system state from outdated observations? When its context is summarized or truncated, how is that information preserved?
If execution is interrupted, can the agent resume without repeating changes that already succeeded? If an early decision later proves incorrect, how does it identify and revisit the actions that depended on it?
What triggers a checkpoint, human review, or stop? Has the vendor compared outcomes across short tasks and operations lasting several hours, and what failures appeared only during longer executions?
5. What happens at scale?
At what scale has the agent actually been tested: how many nodes, concurrent tasks, and agents?
Ask how it tracks dependencies and differences in state across nodes. As the environment grows, how does it preserve the original task and its constraints while processing additional observations?
When multiple agents operate simultaneously, how are conflicting changes prevented? Can one agent detect that another has modified a shared dependency? How are ordering requirements and ownership of changes established?
If part of a deployment succeeds and the rest fails, how is the extent of the failure contained, and how is recovery coordinated? What failure modes appeared only after testing expanded beyond a handful of nodes? What changes were made in response, and what subsequent testing verified those changes?
6. How does your agent handle brownfield?
How has the agent been tested against existing infrastructure with undocumented dependencies, configuration drift, historical exceptions, and partially completed changes?
Ask how it discovers the current state before acting and what evidence it gathers about why a configuration exists. How does it distinguish an error from an intentional exception or a dependency that is not immediately visible?
When the environment contradicts its assumptions, does the agent investigate, request clarification, or proceed? What level of uncertainty causes it to stop, and how does it communicate what remains unknown?
After a change, how does verification establish that the requested task succeeded without disrupting existing behavior? Can the vendor provide results from brownfield environments and compare them with results from clean installations?
7. Do you have a dataset of traces I can examine?
Can the vendor provide the evidence needed to examine its answers to the previous six questions?
Ask for command-level execution traces, with secrets redacted, alongside task instructions, starting conditions, and verification results. Does the dataset include failures, partial completions, and successful runs with unwanted side effects? How were the runs selected, and what proportion of the total evaluation do they represent?
Does each trace identify the model, agent version, available tools, permissions, and execution environment? Does it distinguish container executions from VM executions and identify whether reboot or recreation checks were performed?
Can those traces be used to examine component-specific failures, compare repeated runs, and investigate behavior outside the original scoring criteria? Do they show what the agent claimed, what it executed, and what verification established afterward?
Where execution traces cannot be shared, can the vendor provide reproducible test cases or support an evaluation in an environment you control?
Conclusion
These seven questions provide a starting framework for evaluating how an AI agent will operate in your environment: how it verifies success, handles security and guardrails, behaves across repeated runs, and performs over time, at scale, and on your existing systems.
The answers put the vendor’s success rate in context, showing which results apply to your infrastructure, which depend on specific conditions, and what remains to be tested.
For organizations building their own agents, the same questions can guide the development team’s design and testing, while helping the operations team define acceptance criteria for deployment in its environment.
