AI companies seeking to refine their services for free now face a big obstacle, in the form of Cloudflare.
Cloudflare, which delivers a fifth of the world’s internet traffic, says it is the first internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default.
Starting today, website owners can choose if they want AI crawlers to access their content and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow.
“Cloudflare's new default setting is the first step toward a more sustainable future for both content creators and AI innovators,” said the provider.
For decades, said Cloudflare, the internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. That model is now “broken”, it said, as AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue.
“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare. “Original content is what makes the internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators.”
“Cloudflare’s innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of global published Condé Nast. “This is a critical step toward creating a fair value exchange on the internet that protects creators, supports quality journalism and holds AI companies accountable.”
“AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what,” said Steve Huffman, co-founder and CEO of Reddit. “The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled.”
Cloudflare is enforcing a permission-based model for AI crawlers. AI companies will now be required to obtain explicit permission from a website before scraping. Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access.
This shift means that every new domain starts with the default of control and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can “easily check” their settings and enable crawling at any time if they want their content to be freely accessed, said Cloudflare.
Cloudflare started to allow publishers to block AI crawlers at the back end of last year, but now the process is automatic, with granular control.
Top global publishers and technology companies supporting this permission-based model for AI crawling include: The Associated Press, The Atlantic, BuzzFeed, Condé Nast, Drupal & Acquia, Fortune, O'Reilly Media, Pinterest, Quora, Reddit, Sky News Group, TIME, Universal Music Group, and Ziff Davis, among many others.
