Publishers back a licensing standard for AI web scraping

Called Really Simple Licensing (RSL), the standard builds on robots.txt, which publishers use to tell crawlers what they may access. RSL adds licensing and royalty terms to those files, and lets publishers include terms in books, videos and training datasets. The options include free access, subscriptions, pay-per-crawl charges and pay-per-inference fees when a model references a publisher’s work. Crawlers used for archiving or search indexing can continue as usual.

The RSL Collective, a newly formed rights organization, is led by Eckart Walther, co-creator of RSS and former CardSpring CEO, and Doug Leeds, former CEO of IAC Publishing and Ask.com. Walther told The Verge the aim is to create a scalable business model for the web. The collective says its standard could spare publishers and creators from negotiating individual deals with AI companies. Vox Media, News Corp and The New York Times have already made separate licensing agreements with companies including OpenAI and Amazon.

RSL cannot itself stop bots from visiting a site. The collective is working with content delivery network Fastly on a system to admit AI crawlers only when they have agreed to license content. Publishers using other providers can still request payment, but cannot block crawlers through RSL unless those providers develop a similar tool. AI companies have been accused of disregarding robots.txt instructions, and calculating inference fees would also require their cooperation.

Leeds says collective members would share the cost of enforcing licenses and compares the approach with music rights organization ASCAP. But copyright law around unauthorized scraping and AI training remains unsettled, with major AI companies facing lawsuits from Reddit, Getty Images and online publishers. Leeds and Walther say RSL gives crawlers notice of terms before they reach a site. Joining the collective is free; other supporters include O’Reilly, wikiHow and Ziff Davis, owner of IGN.