Cloudflare defaults to blocking AI scrapers

New domain owners on the platform will now be prompted to decide whether they wish to permit AI scrapers, marking a significant departure from the previous opt-in approach. The initiative includes a pilot "Pay Per Crawl" program, allowing select publishers to set specific fees for AI companies seeking access to their digital archives. By formalizing this barrier, Cloudflare aims to bridge the gap between AI development and intellectual property rights, ensuring companies operate with explicit permission.

This update builds on a series of aggressive countermeasures Cloudflare has deployed since 2023. While early tools relied on the honor system of robots.txt files, the company has since moved to block non-compliant bots entirely. The current strategy also incorporates the "AI Labyrinth" feature, which traps unwanted crawlers in a digital maze to deter scraping. Major entities, including The Atlantic, The Associated Press, and Stack Overflow, have already aligned with these restrictions as the industry grapples with the transition from traditional search engines to AI-driven chatbot responses.