itstargetnews
The viral virtual assistant OpenClaw—formerly known as Moltbotand before Clawdbot—a symbol of a broader revolution It continues to potentially change how the internet works. Instead of being a place where people usually live, the web may soon be dominated by autonomous AI bots.
A new report Measuring bot activity on the web, as well as related data shared by WIRED with internet infrastructure company Akamai, show that AI bots already account for a significant portion of web traffic. The findings also shed light on a more sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to stop them.
“Most of the internet will be bot traffic in the future,” said Toshit Pangrahi, cofounder and CEO of TollBit, a company that tracks web-scraping activity and published the new report. “It’s not just a copyright problem, there’s a new visitor emerging on the internet.”
Most big websites try to limit what content bots can scrape and eat with AI systems training objectives. (WIRED’s parent company, Condé Nast, as well as other publishers, is currently suing several AI companies for alleged copyright infringement related to AI training.)
But another type of AI-related website scraping is now on the rise as well. Many chatbots and other AI tools are now available get real time information from the web and use it to increase and improve their outputs. This might include the latest product prices, movie theater schedules, or summaries of the latest news.
According to data from Akamai, training-related bot traffic has been steadily increasing since July. Meanwhile, global activity from bots that extract web content for AI agents is also growing.
“AI is changing the web as we know it,” Robert Blumofe, Akamai’s chief technology officer, told WIRED “The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the fundamentals of doing business.”
In the fourth quarter of 2025, TollBit estimates that an average of every 50 visits to customers’ websites will come from an AI scraping bot. In the first three months of 2025, that number will be just one in every 200. The company says that in the fourth quarter, more than 13 percent of bot requests bypassed robots.txt, a file that some websites used to indicate which pages bots should avoid. TollBit says the share of AI bots that ignore robots.txt increased 400 percent from the second quarter to the fourth quarter of last year.
TollBit also reported a 336 percent increase in the number of websites attempting to block AI bots last year. Pangrahi says scraping techniques are becoming more sophisticated as sites try to assert control over the way bots can access their content. Some bots disguise themselves by making their traffic appear as if it comes from a normal web browser or send requests designed to mimic how people typically interact with websites. The TollBit study says the behavior of some AI agents is now almost indistinguishable from human web traffic.
TollBit markets tool which website owners can use to charge AI scrapers for accessing their content. Some companies, including Cloudflare, popular similar devices. “Anyone who relies on human web traffic — starting with publishers, but basically everyone — will be affected,” Pangrahi said. “There has to be a faster way to have that machine-to-machine, programmatic exchange of value.”
