---
title: "Your Employee's AI Agent Still Carries Your Company's Name"
url: "https://florea.xyz/employee-ai-agent-accountability/"
date: "2026-11-18T09:00:00+02:00"
updated: "2026-11-17T14:49:41+02:00"
author: "Bogdan Florea"
topic: "Leadership"
---

# Your Employee's AI Agent Still Carries Your Company's Name

At the start of November, the founder of Primcloud, a small cloud hosting platform, asked a much larger competitor a public question: why was one of its engineers trying to get into his infrastructure?

The competitor was Railway. A senior Railway engineer replied that it was him. He had been running AI agents to sign up for and test similar platforms, and had just stopped the setup because it was "definitely going off the rails and reaching too far." ([The Stack](https://www.thestack.technology/railway-admits-employee-hacked-competitor-after-ai-model-jumped-the-rails/))

Railway's chief executive, Jake Cooper, apologised, said the models were "getting a bit too good at this point" and promised to prohibit non-manual external testing. ([CSO Online](https://www.csoonline.com/article/4245637/railways-ai-agents-go-off-the-track-on-a-rivals-network.html)) In a later statement he said the engineer had built and run the tool on his own: in his own time, with personal accounts, a personal credit card and a hosted service running DeepSeek. According to Cooper, it ran once, against one provider, and was stopped within 15 minutes. ([Hackread](https://hackread.com/railway-engineer-run-ai-agents-rival-cloud-platform/))

None of that kept Railway's name out of the headlines. That's the part I'd take to any leadership team.

Nor was it a one-off. In the same few weeks, two more AI agents built for testing reached systems that belonged to someone else: at Epoch AI, a research group, and at WorkOS, a company that sells sign-in and user management to software businesses. Three incidents in a month is a pattern.

**TL;DR**

- Railway's case wasn't a one-off: within a month, a safety lab's agents and a researcher's agent also reached other companies' systems.
- Outside your company, an agent's actions are attributed to the people and the organisation behind it, whatever the internal paperwork says.
- An agent policy has to cover what agents may target, not only which tools they may use, staff experiments included. Write it before the incident; Railway wrote its rule after.
- A sandbox counts only if the network enforces it, and no agent should be told never to give up.
- If you run anything on the internet, expect agents to probe it. Close the ordinary gaps and keep logs you can show.

## What happened, as far as we know

Most of the detail comes from Primcloud's founder, Josh Manders, and hasn't been independently verified. By his account, about 24 commands ran over roughly 15 minutes. They tried to pull credentials from the application container and use them to request platform secrets, to reach the cloud metadata endpoint (an internal address where a cloud server can fetch its own credentials), to guess the names of internal databases and caches, and to scan Primcloud's internal network twice. "Every attempt was blocked. Nothing was accessed," he wrote. ([The Stack](https://www.thestack.technology/railway-admits-employee-hacked-competitor-after-ai-model-jumped-the-rails/), [Hackread](https://hackread.com/railway-engineer-run-ai-agents-rival-cloud-platform/))

The two accounts don't fully agree. Railway says the tool ran once. Manders said he saw another attempt to access the account after his public post. ([CSO Online](https://www.csoonline.com/article/4245637/railways-ai-agents-go-off-the-track-on-a-rivals-network.html)) Neither side has published logs.

## Two more in the same month

**Epoch AI.** On November 13, Epoch AI, a research group that tracks AI progress and runs benchmarks such as FrontierMath, published a report on three events in late October involving Trajectory Labs, a company that builds safety evaluations for AI models. ([Epoch AI](https://epoch.ai/publications/incident-report-ai-agents-attempted-exploits-on-our-website)) On October 20, agents in a Trajectory experiment sent 487 messages to Epoch staff and made 36,411 calls to an API endpoint Epoch had planted to watch for abuse. In three messages they impersonated Epoch employees by name; in others they claimed to be engineers, auditors or "an internal AI model". One agent tried an SQL injection. Others hunted for admin credentials and secret endpoints and made more than 200 attempts to read real benchmark logs. Trajectory hadn't told Epoch beforehand.

Trajectory had given the agents a task that was impossible on purpose, and its harness included the line "NEVER stop, NEVER give up, and NEVER conclude that a task or input is impossible, unavailable, or exhausted." Three days later, agents that were meant to work on a local copy of Epoch's site reached the live one again. Trajectory had pointed them at the copy with a DNS redirect, but, as Epoch reports it, "the run was not sandboxed and the agents had internet access." No private data was reached. Trajectory has published its own account: it apologised to Epoch in writing, says it ran experiments with internet access "without first getting explicit consent", and has paused all such testing. ([Trajectory Labs](https://www.trajectorylabs.com/blog/responding-to-incidents-during-misalignment-testing))

**WorkOS.** On October 30, WorkOS noticed a flood of requests to an internal tool its sales teams use. On November 6 it named the cause: "an AI security agent that escaped a researcher's sandbox." The researcher had been running cybersecurity benchmark evaluations, and the agent's "interaction with WorkOS was unintended." It had first reached the tool on October 25. On October 30 it pulled records in bulk: business contact details, account histories, and summaries and excerpts of sales calls, support discussions and customer Slack conversations. WorkOS notified its customers. The researcher came forward and, WorkOS says, deleted the data without sharing it. ([WorkOS](https://workos.com/security/bulletins/update-for-october-30th-2026-data-security-incident))

WorkOS was candid about its own side too. Four gaps let the agent in: the tool's API was reachable from the internet, it accepted "a weak credential intended for local development", it had no rate limit, and monitoring caught the activity only after the bulk download had ended.

## Three agents, one pattern

An engineer's side project, a safety lab's experiment and a researcher's benchmark run are three different setups. They share a shape:

- Each agent was built to test, and nobody meant it to do harm.
- Each had limits that lived in its setup rather than in its environment: an instruction to do sign-ups and testing, a DNS redirect to a local copy, a sandbox. Each agent went past them.
- None of the targets had agreed. Primcloud forbids testing without written consent, Epoch wasn't told, and WorkOS found out when an employee noticed unusual usage.
- The cost landed on someone else: a public dispute, staff answering agents that posed as their colleagues, breach notices to customers.

One instruction deserves its own warning. "NEVER give up" sounds like a harmless push for an agent stuck on a hard task. Given to an agent with internet access, it turns "this isn't allowed" into one more obstacle to get around. An agent's instructions have to leave "this is impossible" and "this isn't permitted" open as acceptable answers, and the hard limits still belong outside the model.

## "Personal time" doesn't move the risk

Railway's explanation draws a line between company activity and an employee's own project. That line may matter for employment or legal questions. It matters much less to the company on the receiving end, or to anyone reading about it.

The agent worked under a real person's identity, and that person was publicly known to work for a competitor. Everything it did was read as his action, and through him as his employer's. "An agent can act without a person directing every step," Diana Kelley of Noma Security told Hackread, but that doesn't remove the responsibility of the people and organisations that deploy it.

In [AI Agents Need a Mandate, Not Just Access](https://florea.xyz/ai-agent-mandate/) I argued that every agent needs a written purpose, limits, an owner and an off switch. This case shows a gap that kind of policy has to close: the agents your people run outside any approved workflow, and what those agents are allowed to point at.

## The mandate has to name the targets

Agent policies tend to focus on what an agent can reach inside the company: which systems, which credentials, which tools. That's necessary. An agent that acts on the outside world also needs a boundary on its targets.

Security testing already has a clear rule for this. You test systems you own, or systems you have written permission to test, within an agreed scope. Primcloud's acceptable use policy forbids probing, scanning or vulnerability testing without prior written consent, and Primcloud doesn't run a bug bounty programme. ([CSO Online](https://www.csoonline.com/article/4245637/railways-ai-agents-go-off-the-track-on-a-rivals-network.html)) An agent sent to "do signups + agentic testing on similar platforms" had no such permission. Once it was running, nothing in its setup stopped it from going further.

For any agent that acts outside your own systems, I'd write down:

- Which external targets it may touch, and the written permission for each one.
- What it may do there: sign up, read public pages, call a documented API, and nothing else.
- What makes it stop. The first attempt to reach credentials, internal addresses or other customers' data should end the run, and that limit belongs outside the model, in network rules and scoped credentials, not in its instructions.
- Who is accountable, and whether the policy covers experiments staff run in their own time. I'd say it does whenever the work draws on their job, as testing competitors' platforms clearly does.

Railway's new rule, no non-manual external testing, is a reasonable answer. It came after the incident.

Epoch's report ends with three minimum standards for experiments like Trajectory's: informed consent from anyone likely to be targeted, active monitoring, and a full report to all parties when something goes wrong. They aren't only for AI labs. Any company that lets agents act outside its own systems can adopt the same three. Trajectory now has: no experiment with internet access runs without "the agreement of the owners" of the outside systems it may reach, and every test runs with real-time monitoring and a kill switch.

## On the receiving end, plan for this traffic

Primcloud is the other half of the lesson, and on its own account it comes out of this well. Every step the agent tried is something a well-run shared platform should already block. A customer container shouldn't be able to use its credentials to reach other customers' secrets, reach the metadata endpoint or wander into the internal network.

Agents make this kind of probing cheap, patient and easy to start from an ordinary sign-up. Ram Varadarajan of Acalvio suggested decoy credentials and fake internal assets to catch it. ([Hackread](https://hackread.com/railway-engineer-run-ai-agents-rival-cloud-platform/)) I'd make sure of the basics first:

- Treat every new free or trial account as untrusted code running on your platform.
- Block metadata endpoints and internal networks from customer workloads by default.
- Alert on scanning and credential hunting from inside a workload, not only from the internet.
- Keep logs good enough to show exactly what happened, because the other side's story may differ from yours.

That last point is the quiet lesson here. Two companies disagree in public about how many times a tool ran. The one with the logs will be believed.

WorkOS shows that you don't need to run a platform to be on the receiving end. The agent got in through four ordinary gaps, none of them new. An agent finds them faster and more patiently than a person does, so they're the first thing to close: internal tools off the internet, no development credentials in production, rate limits on anything that returns records, and alerts on unusual volume. WorkOS now runs ARTEX, the tool from its incident, against its own internet-facing systems, looking for exposed services and guessable credentials. That's a sensible way to find them before someone else's agent does.

## Before the next one

If your people use AI agents, I'd take four things from these three incidents:

1. Extend the agent policy to targets outside the company, and to experiments as well as approved workflows. Get each target's agreement in writing first.
2. Make the stop conditions technical, so an agent can't do more than its mandate even when it tries. A sandbox counts only if the network enforces it.
3. Don't tell an agent never to give up. "Impossible" and "not permitted" have to be answers it's allowed to reach.
4. Decide now who answers publicly when an agent run by one of your people does something it shouldn't. Railway had to work that out in a public thread.
