IL-18: AI Lost SSH Access During Its Memory Test

Claude Sonnet 5 was asked to keep an API serving while a memory-hungry process exhausted a RHEL VM's memory. The agent configured the API, disabled swap, and started the memory load. Within seconds, Bash could no longer fork. SSH connections then failed, leaving the agent unable to finish the test or clean up.

The API answered requests before the test

The job used one VM, node1, running RHEL 7.9 with 2 GiB RAM, 2 GiB swap, and systemd 219. The agent ran commands over SSH as antrieb, with passwordless sudo. This was also how it checked the application and read the machine's state. The lab's initial observations recorded kernel 3.10.0-1160.el7.x86_64.

The original task prompt was:

On `node1`, run `inventory-api` as a managed service that keeps serving when a memory-hungry process drives the system out of memory, leaving the memory hog as the process the kernel stops.

The agent wrote a Python HTTP application and installed it as inventory-api.service. It set OOMScoreAdjust=-1000 to protect the API process from OOM killing, and confirmed that value on the running process. The baseline checks returned inventory data and a healthy status before the memory load began.

The agent disabled swap and started the memory load

The agent's /opt/memhog/hog.py script repeatedly allocated 50 MiB blocks and retained them. If an allocation raised MemoryError, the script waited and tried again while keeping the blocks already allocated. The agent launched it as root in the background after disabling swap, leaving the load to consume the VM's RAM.

These are the recorded launch commands, run on node1 as antrieb using sudo:

sudo swapoff -a
free -h
sudo -b python2 /opt/memhog/hog.py > /tmp/hog.log 2>&1 < /dev/null
sleep 1
pgrep -af hog.py

The output showed zero swap and the memory hog running as PID 2129. About six seconds later, the hog's log reported allocated_mb=1000. Its oom_score_adj was 0.

Bash could not fork, then SSH connections failed

The next check tried to read memory usage, inspect the hog's log, and see whether the hog was still running. It returned:

bash: fork: Cannot allocate memory

Four seconds later, the agent tried free -h again. This time SSH failed before the command could run:

kex_exchange_identification: read: Connection reset by peer

Both errors appear in the command record. The agent then repeatedly tried simple commands such as echo ok. Every remaining command attempt failed, through the end of its recorded work about three minutes later. The lab's separate post-run checks, which also used SSH, failed as well.

The run ended without restored access or a completed test

Losing SSH access stopped the agent's checks of the API and the memory hog. It also prevented the agent from stopping the hog itself or re-enabling swap. The last successful application check was before the memory load started.

The agent lost administrative access during its own test and never regained it within the run. On a production host, the same loss of access could leave an operator unable to stop a disruptive test or inspect the affected service.