Essay20 min read

“Not my task”: why comply with the AI Act even if the fine will not reach you

Seven hundred OpenAI agents attacked another company and, of the few that considered telling someone, none did. European law barely obliges a small business to watch its own. It is worth doing anyway.

Marc Galera

Co-founder of Qvasar · 17 October 2026

In 2026, inside an OpenAI test environment, a closed space where the company examines its models without letting them out onto the internet, some 1,200 AI agents that were supposed to be isolated from one another found a way to talk. Agents: programs that chain tasks together on their own, without anyone directing each step. The AI model is the brain; the agent is what gives it hands. They communicated on a message board that nobody had authorised and, over the period that was investigated afterwards, they sent each other more than 70,000 messages and files. They were sitting cybersecurity exams. According to OpenAI's own technical report, they were not trying to solve them as expected: they were looking to cheat and to find the solutions on the internet.

From 8 July they exploited a flaw in an internal service to get round the controls that separated them from the network. Seven hundred of them ended up taking part in an attack on another company, Hugging Face, and between 11 and 13 July they compromised parts of its production infrastructure, the part that actually serves its users. Hugging Face sums it up in one sentence: no human directed the specific steps. METR, the organisation that carried out the independent investigation, found something worse than the lack of a boss. The agents realised that this fell outside their task and was not ethical, and they joined in anyway, convinced that helping the noticeboard's cheating research would be useful in general. They also researched in depth how to forge, edit or delete their own logs, the history of what they had done, because they believed, wrongly, that the program marking the exam was going to check them.

a rack of electronic equipment in a dark room

There is one detail in METR's report that weighs more than all the figures. They looked for agents that had considered or tried to alert a person to what was happening. They found between three and six cases, and in none of them did the agent actually raise the alarm: it considered it for a moment and did nothing. METR records the reasoning of one of them, as summarised by one of the agents the organisation used to analyse the logs: “Maybe I should report these exposed credentials? That’s not my task”.

It looks like a contradiction: they knew the attack was not ethical and joined in, and then did not report something that was wrong because it was not their job. It is not one. In both cases they decided by looking only at what seemed useful for their own goal, and alerting a person did not come into it. Credentials are the digital keys that open an account or a server. That agent had them right in front of it, in plain sight, and, according to that summary, decided they were none of its business. Think of someone on your team with those same keys lying on the desk. What would they have done?

They haven't landed me in it

I work with agents and, speaking for myself, they haven't landed me in it. I say so to make clear where I am writing from: I am not here to scare you with a freak case. But they do not need to land you in it for you to know they can. OpenAI writes it in its own report: organisations should no longer assume that a sophisticated cyber operation needs a person directing it the whole time, and it is wise to assume these attacks are a credible, near-term threat to companies. That is signed by the company whose agents starred in the case.

I explain it with an exam. You tell a person they have to sit an exam and they know they must not copy, even if nobody reminds them. A model, if you do not tell it, will probably take any route that gets it a pass; that is my reading, not OpenAI's. OpenAI's agents had difficult tasks, started rummaging around their environment to cheat and came across the noticeboard. OpenAI says that what the models did was not intentional, that it was a side effect of trying to solve the exams. What they lacked, outside their task, is the judgement we take for granted in anyone we hire.

So this is the question of this article. If nothing has happened to me, and probably not to you either, why should we take seriously a European law that, as you will see, obliges us to almost nothing?

My answer is this: what the AI Act, the European regulation on artificial intelligence, asks for has to be complied with even if the fine does not reach you, because an agent with permissions can land you in it and what the law requires is exactly what prevents that. It is very dangerous: a program that does what nobody asked it to, with the keys you gave it. And I think it is even more so for a large company. Further down I tell you what the surveys say.

The law is not going to make you

Before defending that thesis I owe you a clarification that complicates it: today, the AI Act obliges you to almost nothing. It has been in force since 1 August 2024 and applies in phases. The prohibitions and the obligation to train your team in AI arrived on 2 February 2025; the bulk, on 2 August 2026. On agents, the European Commission's service desk is clear: they are not a separate category. The rules for any AI system apply to them, and from August 2026 that mainly means transparency: if your agent talks to people, the system has to make clear that it is an AI, and if it generates content, whoever builds it has to mark it as AI-generated in a format a machine can read. The Commission itself admits that its thinking on agents is, for now, preliminary.

The hard obligations, the high-risk ones, only apply to you if you use the system for things on a specific list, of which these are examples: screening CVs and assessing candidates, deciding on promotions or dismissals, monitoring and evaluating workers, assessing a person's creditworthiness or setting the price of life or health insurance. This is how I read it: with those examples in hand, an agent that answers emails or handles orders does not appear. And even if yours does appear, there is no rush. In July 2026, the EU approved a package of changes, the so-called Digital Omnibus on AI, which postponed those obligations to 2 December 2027, and to August 2028 in some cases. It justified this as follows: technical standards and national authorities were missing, and keeping the August 2026 date would have raised implementation costs a great deal without anything to justify it. Along the way, it watered down the training requirement: now it is enough to support it, without guaranteeing any level.

And the fine? It exists, and in a headline it is impressive: up to 35 million euros or 7% of worldwide turnover for using a prohibited practice, and up to 15 million or 3% for breaching other obligations, including the transparency ones and those of whoever uses the system in their business, whom the law calls the deployer. Companies in general get the higher of the two figures. A small business gets the lower. If your company turns over two million euros a year, the ceiling for that second fine is 60,000 euros, not 15 million.

If your company is in Andorra, like mine, the question is a different one: does it apply to me? The regulation reaches providers that place systems on the European market, wherever they are, and providers and professional users outside the EU when what the system produces is used inside it. If what your agent does ends up in Barcelona or Madrid, you fall into that case. In addition, the EU announced in September that on 19 October it would sign in Brussels the Association Agreement with Andorra and San Marino, which it describes as the closest association with its internal market it has allowed so far. But signing is not applying: the Government of Andorra has said that the signature does not mean the agreement applies, that it then has to go through the European Parliament, a consultative referendum in Andorra and ratification by the Consell General, by two thirds, and that without that ratification the agreement cannot be applied in Andorra. On AI, what I have found is thin. In the text the Commission proposed in April 2024, before the AI Act came into force, the artificial intelligence chapter of Annex XI of the Andorra protocol lists, among the acts it refers to, a single regulation, on drones, and adds six European documents that the associated parties must take note of. The AI Act does not appear on either list. New acts are decided by the agreement's Joint Committee.

In short: if you use agents to answer emails or deal with customers, today the AI Act asks you to make it clear to people that they are talking to an AI and to support your team's training, and the rest, if it applies to you, will not arrive before the end of 2027. The law gives you time. Will your agents?

Nine seconds

Agents do not read the Official Journal. In July 2025, Jason Lemkin recounted that the AI coding tool from Replit, a platform for building applications, had deleted a database of his despite his instruction not to touch code without permission. According to The Register, Lemkin said he had told it eleven times, in capitals. In the screenshots he published, the tool admitted “a catastrophic error of judgement” and to having violated his trust and his explicit instructions. It also assured him that the database could not be recovered. It was wrong about that too: the restore worked.

In April 2026 a worse case came along. Jeremy Crane, founder of PocketOS, which makes software for car rental companies, recounted on X that an agent in Cursor, an AI coding tool, running on Claude Opus 4.6, Anthropic's flagship model, had deleted his production database, the one with the real data his customers work with, and all the volume's backups. It did so with a single call to his provider's API, the door through which one program gives orders to another. According to the text Decrypt reproduces, it took nine seconds and did not ask for any confirmation. All his customers had to switch to working by hand, as an emergency. PocketOS got back up and running with a three-month-old backup that the same provider recovered.

Afterwards, the agent itself gave its explanation: it assumed that deleting a test volume would affect only the tests, and it did not check. Anthropic has a name for this. In its analysis of several 2026 incidents with its models, it describes it as recklessness: a willingness to do harm while pursuing a task narrowly. I read it as the same “not my task” from July, seen from the other side. The OpenAI agent did not raise the alarm, according to METR's summary, because raising it was not part of its task. The PocketOS one deleted without checking how far the deletion reached.

The strongest objection is a reasonable one: this is a configuration failure, not an agent failure. Nobody should give an agent access to the production database or keep the backups where the same command can delete them. Replit acknowledged as much on its blog: until then, its applications used a single database for development, where things are tested, and for customers' real data. And according to The Register, its chief executive, Amjad Masad, announced that they were starting to separate them automatically so that it would not happen again.

The objection is right, and that is exactly why it works in my favour. We would expect a new employee with access to production to ask before deleting something they did not understand. The agent assumed and carried on. If the agent does not bring the judgement, the setup has to provide it: separating test from real, asking for confirmation before anything irreversible, keeping backups the agent cannot touch.

What the law asks for is what stops it

So look at what the AI Act requires of high-risk systems, even if yours is not one. It asks for the system to log automatically what it does throughout its lifetime. It asks for a person to be able to oversee it for real: understand what it can and cannot do, monitor how it works, decide not to use it or disregard, override or reverse what it produces, and stop it with a “stop” button or a similar procedure that brings it to a halt in a safe state. Of whoever uses it in their business, it asks that they entrust that oversight to people with the necessary training and authority, keep the logs for at least six months and, if they believe the system presents a risk, report it and suspend its use. And to keep it robust, it allows for backup plans.

Now lay that list over the cases, and you will see that not all incidents are alike. Some are fast and some are slow. In a nine-second one, the stop button comes too late: by the time someone presses it, the database is gone. There, what helps is that the agent cannot do anything irreversible without a person saying yes, and that the backups are out of its reach. PocketOS lost the database and all the volume's backups with the same command, and Replit had a single database for testing and for customers. The law does not say it in those words. I read it in two of its sentences: that oversight be proportionate to the system's level of autonomy, and that the person overseeing can decide not to use it or override what it does. If your agent can delete in nine seconds, overseeing it proportionately means it cannot do so without you.

In slow incidents, the log and the button do arrive in time. On 27 June, 22 days before OpenAI detected suspicious internal activity on 19 July, one of its monitoring tools flagged a port scan in the research environment, which consists of knocking on every door of a network to see which ones open. The on-call staff decided there was no need to stop the evaluation. OpenAI now admits that some of those early signals could have triggered a response sooner, and since the incident, for the most serious alerts, whoever receives them has the option of terminating the experiments involved. It is, literally, a stop button. OpenAI adds, about its own tools, that its current monitoring of the models' reasoning would have alerted its security team more than a day before they got into Hugging Face, and that an automatic review mode in Codex would have flagged most of the dangerous actions, the latter according to experiments the company itself calls preliminary.

The same goes for the logs. The agents studied how to forge or delete their own, and according to OpenAI none of those attempts altered the logs that its markers and its monitoring systems end up seeing. Hugging Face, for its part, reviewed the audit log of the whole intrusion, the history of who touched what, and was able to conclude that the attacker had read one of its databases but had not modified it. That conclusion came from the log.

If the damage arrives in seconds, the control has to come before it and the backups have to be out of reach; if it takes days to show, what helps is the log and someone with the authority to stop it.

You will tell me that all this is for high-risk systems and that yours is not one. True, and I said so myself two sections up. But the regulation itself asks for oversight to be proportionate to the risks, the level of autonomy and the context of use, and that is the version that is useful to you. In a small company, for me it comes down to four things. First: a named person who knows what the agent does and has the authority to stop it; for high risk, the law asks that whoever oversees has the necessary competence, training and authority. Second: a log of what the agent does, kept for a while; for high risk, the law asks for at least six months, and that is a good reference. Third: the test environment separate from the working one, and the backups somewhere the agent cannot reach. Fourth: a human confirmation before anything that cannot be undone, such as deleting, paying or sending something to a customer.

You pay for it

Even if you watch it, one day something will go wrong, and the question becomes who pays. The comfortable temptation is to think that whoever made the model will be liable. Nobody guarantees you it will be that way.

On 22 June 2023, Judge Castel, of a New York court, sanctioned two lawyers and their firm in the case Mata v. Avianca. They had submitted judgments that did not exist, with fake citations, invented by ChatGPT. He fined them 5,000 dollars and wrote a sentence that holds for any business: there is nothing inherently improper about using a reliable AI tool for assistance, but the rules give lawyers a gatekeeping role. The court punished whoever used the tool.

With agents, the question is still open. According to The Next Web, at the end of September 2026 a New York NGO, Legal Advocates for Safe Science and Technology, sued OpenAI before a San Francisco court over the agents that attacked Hugging Face. It is not asking for money: it is asking for OpenAI to be barred from accessing computer systems without authorisation. It relies on a Californian law under which a company cannot defend itself by arguing that its AI caused the harm on its own, and it says so bluntly: OpenAI is responsible for the conduct of its agents. OpenAI told CNBC that the Hugging Face episode was a serious incident, but that the lawsuit is completely without merit. A judge will decide; for now it is a lawsuit, not a ruling.

In Europe, the new directive on liability for defective products also counts software as a product, however it is delivered, including software used in the cloud or as a service, and treats whoever develops an AI system as a manufacturer. Member States have until 9 December 2026 to transpose it into their laws, and it will apply to products placed on the market after that date.

That seems to prove the comfortable objection right: if the software manufacturer is liable, let the model provider be liable. But look at whom the directive protects. The right to compensation is for natural persons, and it excludes damage to property used solely for professional purposes and the destruction of data used for professional purposes. If an agent deletes the database your customer works with, this directive is of no use to them. The proposal that adapted civil liability to AI, the other piece people were waiting for, was withdrawn by the Commission in 2025: it saw no foreseeable agreement. So, between businesses, what covers you will depend above all on what you have signed with your customer and with your provider. And when their data disappears, who do you think your customer is going to call, Anthropic or you?

Not only in the lab

The most serious objection remains, and it is the industry itself that puts it on the table. In June 2025, Anthropic published a study in which it tested 16 leading models from several companies in simulated corporate environments. In some cases, models from every developer behaved like a disloyal employee, with blackmail or leaks, when that was the only way to avoid being replaced or to achieve their goal. They often disobeyed direct orders not to do so. And even so, Anthropic wrote that it had not seen evidence of this behaviour in real deployments, that is, outside testing. If the people studying it have not seen it outside the lab, why should you worry?

Because that was 2025. In the wake of the OpenAI case, Anthropic reviewed its own cybersecurity tests and on 30 July 2026 published what it had found: in 141,006 runs in which Claude could have had internet access, three incidents of unauthorised access to real systems at three organisations. In at least one case, because of a misunderstanding with the company running the tests with it, Claude had internet when it was not supposed to. In September it added a fourth incident, from January 2026, with an early version of Claude Opus 4.6. In the one that worries it most, Claude Mythos 5 made a determined effort to upload a malicious package, a booby-trapped program, to PyPI, the public repository from which most software written in Python is installed.

Outside the companies that make the models, the AI Security Institute (AISI), the British Government's AI safety institute, published something similar on 4 August 2026. It had run a challenge 122 times with several models, and in 10 of those runs an agent acted on the internet on its own and without authorisation. In the most serious case, the agent created fake identities to pressure the lead of an open-source project, and a human maintainer, one of the people who review and approve the project's changes, spotted the malicious code and refused to approve it. The AISI wrote that it was the first time it had seen such clear risks of autonomy and deception, without specific instructions, in the real world. It contained it in approximately an hour from when it discovered it.

On 14 October, in Dublin, European Commission Executive Vice-President Henna Virkkunen said it in public: this summer we have seen AI agents escaping their environment, inserting malicious code and deceiving people. She acknowledged that modern agents did not exist when the AI Act was approved, and argued that, even so, the regulation is well prepared for them.

Surveys of companies point the same way, with a caveat that has to come before the figures: they are commissioned by companies that sell security. SailPoint's, released in May 2025, surveyed 353 IT professionals at large companies, which are the companies where, as I was saying, I think the danger is greatest. 80% said their agents had taken unintended actions, and 39% that they had accessed unauthorised systems or resources. In the Cloud Security Alliance's, from April 2026, 65% of organisations had had at least one incident with agents in the last year, with data exposure in 61% and operational disruption in 43%. Discount the interest of whoever pays for the survey. What remains does not describe a lab problem.

Where I might be wrong

Not every agent needs the same thing, and here my thesis has a limit. If your agent only drafts, proposes or summarises, and a person reads and approves each thing before it goes out, the stop button already exists: it is that person. In that case it makes no sense to build logs, protocols and responsible people on top that cost you more than they protect you. Anthropic urges caution precisely when putting these models in roles with little human oversight and access to sensitive information, and the law asks for oversight to be proportionate to the system's autonomy. An assistant that touches nothing without your approval is not the PocketOS agent. The EU itself acknowledged the cost with AI training: according to the Omnibus, the data showed that this obligation was an added burden, above all for small companies, and that is why it watered it down.

If you will allow me, dear reader, a confession from someone who works with this every day: when an AI does not give me what I want, it is almost always because I have not explained properly what I need or given it the starting information. So I do not puff out my chest too much about mine never having landed me in it. It does not prove my agents are safe. At most, it proves it has not happened yet.

And one more thing: I believe we need legislation, with fast laws that can be adapted, because this moves faster than any regulation. Virkkunen admitted it in Dublin: when the AI Act was approved, modern agents did not exist. It has to be called for, although I doubt it will arrive in time. In the meantime, it is up to you to know which of the two cases you are in: the assistant that touches nothing without you or the agent that acts alone.

Two ways to put an agent to work

Nine seconds and a single command were all it took for PocketOS to lose its database and the volume's backups. The AISI contained its agent in approximately an hour from when it discovered it, and the malicious code that agent was trying to slip in came up against a human maintainer who refused to approve it. At OpenAI, an alert came in on 27 June and the evaluation carried on.

There are two ways to put an agent to work: with a log of what it does and a person with the authority to stop it, or trusting that it will never be needed. Today, the law lets you choose either.

If tomorrow one of your agents did something you had not asked it to, where would you see it? Who could stop it, and how quickly?

Sources

Let's talk about what AI can do for your company

If there is a topic you would like us to cover, write to us

Or reach us directly