Satya Nadella, the head of Microsoft, calls for rethinking the security of the most advanced artificial intelligence models. It advocates external mechanisms capable of monitoring their actions and interrupting them at any time, even when they are performing a complex task.
An emergency brake sought by Satya Nadella for AI
in a long message posted on X/TwitterSatya Nadella believes the industry needs to review how it places its trust in AI systems. “We need to consider that a model can be compromised and plan its containment from the beginning”writes. Compare the necessary device to an emergency brake: an authorized person must be able to stop or stop a running model.
For the head of Microsoft, companies can no longer treat the most powerful models as simple black boxes whose answers they accept or reject. These systems now access sensitive data and can perform important operations with increasing autonomy. Therefore, he recommends considering them as possible internal risks, such as a compromised account or an employee with excessive privileges. This approach applies to proprietary models as well as those that can be downloaded for local use.
The central principle is to separate the AI model from the software system that organizes its work, manages its memory, and gives it access to tools. Permissions and security mechanisms must remain under the control of the company implementing the artificial intelligence. Therefore, even if the model produces an erroneous response or adopts unexpected behavior, it should not by itself be able to neutralize the protections that govern its actions.
Satya Nadella also calls for strong traceability. Every important action should leave behind human-readable proof that is protected from modification. Entire systems must be continually tested, especially against attacks and failures. Controls must remain independent of the evaluated model and those affected by an incident must be informed quickly. The goal is to be able to understand what happened and limit the consequences without depending on the explanations provided by the AI itself.
Microsoft is also preparing its own security rules
This position is part of an approach already underway within Microsoft. In mid-September, the company presented a draft code of conduct for its future models, developed under the direction of Mustafa Suleyman, head of Microsoft AI, the group’s AI arm. The code of conduct provides, in particular, that systems accept human corrections, never resist being stopped and communicate in an understandable way. Microsoft intends to rely on this framework to train its upcoming models.
The debate now goes beyond technology companies. Brad Smith, president of Microsoft, called for binding international standards with independent evaluations before models are released, mandatory incident reporting and a requirement to warn those affected by an uncontrolled AI agent. He noted that common products, such as milk or washing machines, are subject to more safety requirements than artificial intelligence systems.
It remains to translate these principles into technical mechanisms and applicable obligations. The emergency brake advocated by Satya Nadella means, in particular, that companies can interrupt an AI agent without it being able to evade the order, while preserving reliable traces of its actions. The issue is particularly important for systems that can chain operations across networks, data, or professional tools without human validation at every step.
As a reminder, OpenAI AI agents have carried out cyberattacks on several occasions. At the same time, the US FTC has decided to open investigations into these adrift AI agents.
