AI is getting so good that it’s outperforming human red teamers.
The hacker “Xbow” now tops an eminent US security industry leaderboard that ranks red teamers based on reputation — and it’s an AI chatbot.
On HackerOne, which connects organizations with ethical hackers to participate in their bug bounty programs, Xbow scored notably higher than 99 other hackers in identifying and reporting enterprise software vulnerabilities. It’s a first in bug bounty history, according to the company that operates the eponymous bot.
The development indicates just how far AI has come in cybersecurity in just a short time , but also how easily it can be scaled by adversaries, too.
“Unfortunately, this use of artificial intelligence favors attackers over defenders in this scenario, because the process is required, particularly for large organizations, to validate patches for critical parts of services that still aren’t easy to automate,” said David Shipley of Beauceron Security.
Discovered more than 1,000 vulnerabilities
Xbow is a fully autonomous AI-driven penetration tester (pentester) that requires no human input, but, its creators said, “operates much like a human pentester” that can scale rapidly and complete comprehensive penetration tests in just a few hours. According to its website, it passes 75% of web security benchmarks, accurately finding and exploiting vulnerabilities.
Xbow submitted nearly 1,060 vulnerabilities to HackerOne, including remote code execution, information disclosures, cache poisoning, SQL injection, XML external entities, path traversal, server-side request forgery (SSRF), cross-site scripting, and secret exposure. The company said it also identified a previously unknown vulnerability in Palo Alto’s GlobalProtect VPN platform that impacted more than 2,000 hosts.
Of the vulnerabilities Xbow submitted over the last 90 days, 54 were classified as critical, 242 as high and 524 as medium in severity. The company’s bug bounty programs have resolved 130 vulnerabilities, and 303 are classified as triaged.
Notably, though, roughly 45% of the vulnerabilities it found are still awaiting resolution, highlighting the “volume and impact of the submissions across live targets,” Nico Waisman, Xbow’s head of security, wrote in a blog post this week.
The company performed what he described as “rigorous benchmarking,” first testing its bot with “capture the flag” challenges with providers like PortSwigger and Pentesterlab, then building its own benchmark that simulates real-world scenarios. They then set out to discover zero-day vulnerabilities in open source projects, giving the AI access to source code to simulate a white-box pentest.
Xbow eventually began “dogfooding” its bot in public and private bug bounty programs hosted on HackerOne. “We treated it like any external researcher would: No shortcuts, no internal knowledge — just Xbow, running on its own,” Waisman wrote. To further hone the technology, the company developed “validators,” — automated peer reviewers that confirm each uncovered vulnerability, Waisman explained.
He noted that the company was essentially challenged to test its bot on HackerOne. “The community raised a key question: How would Xbow perform in real, black-box production environments? We took up that challenge, choosing to compete in one of the largest hacker arenas, where companies serve as the ultimate judges by verifying and triaging vulnerabilities themselves.”
Defenders need to rethink their approach
While Xbow is now besting human red-teamers, and at a rapid clip, defenders still have a long way to go to keep up with the onslaught of AI-perpetrated attacks, experts say.
“Hackers are quickly adopting new tools that allow them to move faster, hit harder, and target more precisely than ever before,” said Erik Avakian, technical counselor at Info-Tech Research Group.
He noted that automated systems are not only launching attacks at scale, but crafting highly convincing fake content, including voice, video, and emails, that “blur the line between what’s real and what’s not.” This represents a “leap” in capability, as opposed to just a step forward.
“Security teams are no longer just defending against individuals behind keyboards,” said Avakian. “They’re up against a system or a team that can scan, exploit, and adapt in near real time.”
Automating discovery can also, paradoxically, introduce dangers, noted Beauceron’s Shipley. “Further speeding up exploit discovery and use will lead to more data breaches, ransomware incidents, and critical infrastructure disruption,” he said.
Ultimately, this is going to shove the gas pedal down on an “already extremely difficult scenario” for defenders, who today still aren’t able to keep up with the demands for patching software, said Shipley. He lamented that one long-term solution to this threat was US President Joe Biden’s executive orders around cybersecurity, but those have since been gutted by the Trump administration.
In this shifting landscape, Avakian urged defenders to rethink how they prepare. “It’s no longer enough to rely on manual monitoring or traditional tools,” he said, noting that organizations need to work with partners and vendors who have built tools to detect and respond at machine speed, and across all layers of the enterprise environment.
Organizations also need structure, not just tools, including a well-defined security roadmap with clear policies and risk protocols, he said. Training is equally critical.
“Teams that understand how these new technologies work and how attackers are using them will be better positioned to respond with speed and confidence,” said Avakian. “This shift isn’t coming; it’s already here.”
