(11 Feb 2025) Web publishers have responded to AI with a trifecta of lawsuits, legislation, and computer science. What began with a litany of copyright infringement suits, including one from the New York Times, has turned into a wave of restrictions on use of websites’ data, as well as legislation such as the EU AI Act to protect copyright holders’ ability to opt out of AI training.
However, legal and legislative verdicts could take years, while the consequences of AI adoption are immediate. So in the meantime, data creators have focused on tightening the data faucet at the source: web crawlers. Since mid-2023, websites have erected crawler restrictions to over 25% of the highest-quality data. Yet many of these restrictions can be simply ignored, and while major AI developers like OpenAI and Anthropic do claim to respect websites’ restrictions, they’ve been accused of ignoring them or aggressively overwhelming websites (the major technical support forum iFixit is among those making such allegations).
Now websites are turning to their last alternative: anti-crawling technologies. A plethora of new startups (TollBit, ScalePost, etc), and web infrastructure companies like Cloudflare (estimated to support 20% of global web traffic), have begun to offer tools to detect, block, and charge nonhuman traffic. These tools erect obstacles that make sites harder to navigate or require crawlers to register.
These measures still offer immediate protection. After all, AI companies can’t use what they can’t obtain, regardless of how courts rule on copyright and fair use. But the effect is that large web publishers, forums, and sites are often raising the drawbridge to all crawlers—even those that pose no threat. This is even the case once they ink lucrative deals with AI companies that want to preserve exclusivity over that data. Ultimately, the web is being subdivided into territories where fewer crawlers are welcome.
Read more from MIT Technology Review here.
