The flash crash is probably the most well-known example of the dangers raised by agents—automated systems that have the power to take actions in the real world, without human oversight. That power is the source of their value; the agents that supercharged the flash crash, for example, could trade far faster than any human. But it’s also why they can cause so much mischief. “The great paradox of agents is that the very thing that makes them useful—that they’re able to accomplish a range of tasks—involves giving away control,” says Iason Gabriel, a senior staff research scientist at Google DeepMind who focuses on AI ethics.
“If we continue on the current path … we are basically playing Russian roulette with humanity.”
Yoshua Bengio, professor of computer science, University of Montreal
Agents are already everywhere—and have been for many decades. Your thermostat is an agent: It automatically turns the heater on or off to keep your house at a specific temperature. So are antivirus software and Roombas. Like high-frequency traders, which are programmed to buy or sell in response to market conditions, these agents are all built to carry out specific tasks by following prescribed rules. Even agents that are more sophisticated, such as Siri and self-driving cars, follow prewritten rules when performing many of their actions.
But in recent months, a new class of agents has arrived on the scene: ones built using large language models. Operator, an agent from OpenAI, can autonomously navigate a browser to order groceries or make dinner reservations. Systems like Claude Code and Cursor’s Chat feature can modify entire code bases with a single command. Manus, a viral agent from the Chinese startup Butterfly Effect, can build and deploy websites with little human supervision. Any action that can be captured by text—from playing a video game using written commands to running a social media account—is potentially within the purview of this type of system.
LLM agents don’t have much of a track record yet, but to hear CEOs tell it, they will transform the economy—and soon. OpenAI CEO Sam Altman says agents might “join the workforce” this year, and Salesforce CEO Marc Benioff is aggressively promoting Agentforce, a platform that allows businesses to tailor agents to their own purposes. The US Department of Defense recently signed a contract with Scale AI to design and test agents for military use.
Scholars, too, are taking agents seriously. “Agents are the next frontier,” says Dawn Song, a professor of electrical engineering and computer science at the University of California, Berkeley. But, she says, “in order for us to really benefit from AI, to actually [use it to] solve complex problems, we need to figure out how to make them work safely and securely.”
PATRICK LEGER
That’s a tall order. Like chatbot LLMs, agents can be chaotic and unpredictable. In the near future, an agent with access to your bank account could help you manage your budget, but it might also spend all your savings or leak your information to a hacker. An agent that manages your social media accounts could alleviate some of the drudgery of maintaining an online presence, but it might also disseminate falsehoods or spout abuse at other users.
Yoshua Bengio, a professor of computer science at the University of Montreal and one of the so-called “godfathers of AI,” is among those concerned about such risks. What worries him most of all, though, is the possibility that LLMs could develop their own priorities and intentions—and then act on them, using their real-world abilities. An LLM trapped in a chat window can’t do much without human assistance. But a powerful AI agent could potentially duplicate itself, override safeguards, or prevent itself from being shut down. From there, it might do whatever it wanted.
As of now, there’s no foolproof way to guarantee that agents will act as their developers intend or to prevent malicious actors from misusing them. And though researchers like Bengio are working hard to develop new safety mechanisms, they may not be able to keep up with the rapid expansion of agents’ powers. “If we continue on the current path of building agentic systems,” Bengio says, “we are basically playing Russian roulette with humanity.”
