Cloudflare Lets Sites Charge AI Bots With ‘Pay Per Crawl’ - 1

Image by MArco Verch, from CCnull

Cloudflare Lets Sites Charge AI Bots With ‘Pay Per Crawl’

  • Written by Kiara Fabbri Former Tech News Writer
  • Fact-Checked by Sarah Frazier Former Content Manager

Cloudflare is giving publishers a new option: charge AI bots for crawling their sites using a revived HTTP 402 payment system.

In a rush? Here are the quick facts:

  • Publishers can now allow, block, or charge crawlers per visit.
  • System uses HTTP 402 code to request payments from bots.
  • Cloudflare manages payments and infrastructure for the service.

Through its new “ Pay Per Crawl ” system website owners can charge AI bots to access their content. The system provides publishers with an alternative approach from their current two-option policy which either allows AI data scraping, or complete blocking of AI access.

“Many publishers, content creators and website owners currently feel like they have a binary choice — either leave the front door wide open for AI to consume everything they create, or create their own walled garden,” Cloudflare said. “But what if there was another way?”

With Pay Per Crawl, content creators can now decide who gets in and at what price. They can let some AI crawlers in for free, block others entirely, or charge for access. “We wanted content creators to have control over who accesses their work,” Cloudflare said. “Creators should be in the driver’s seat.”

The move comes amid growing backlash over AI companies using web content without consent. For example, YouTube has been criticised for allowing Google to video scrape without notifying creators . Additionally, Google’s AI Overviews feature has reduced traffic to news sites like the HuffPost and The Washington Post by more than 50%.

“Google just takes content by force and uses it with no return — the definition of theft,” said Danielle Coffey, president of the News/Media Alliance.

The tool works using an old web feature: HTTP response code 402, which stands for “Payment Required.” If an AI bot tries to access a page, the server can now reply with a 402 and include a price tag. If the bot agrees to pay, the server delivers the content.

Cloudflare provides the technical infrastructure and handles payments. Publishers can set a fixed price per crawl request and even apply different rules for different bots. Even if a bot isn’t registered with Cloudflare, it can still be “charged” — essentially blocking access, but leaving room for future deals.

This system could lead to more flexible licensing and dynamic pricing in the future. As Cloudflare puts it: “By providing creators with a robust, programmatic mechanism for valuing and controlling their digital assets, we empower them to continue creating the rich, diverse content that makes the Internet invaluable.”

Data-Labeling Firm Surge AI Aims To Raise $1 Billion In Capital - 2

Photo by Mario Gogh on Unsplash

Data-Labeling Firm Surge AI Aims To Raise $1 Billion In Capital

  • Written by Andrea Miliani Former Tech News Expert
  • Fact-Checked by Sarah Frazier Former Content Manager

The AI data-labeling company Surge AI aims to raise $1 billion in its first capital-raising round. The startup expects to capitalize on rising demand after Meta invested in Scale AI, its competitor.

In a rush? Here are the quick facts:

  • Surge AI aims to raise $1 billion during the company’s first capital raising round.
  • The data-labeling company targets a $15 billion valuation.
  • OpenAI and Google are turning to Surge AI after Meta partnered with Scale AI.

According to Reuters , anonymous sources explained that Surge AI hired advisors to negotiate the deal and is targeting a valuation of $15 billion. The discussions are at an early stage, but the company remains optimistic as Scale AI lost customers after Meta’s acquisition , who are now interested in Surge AI—including OpenAI and Google.

Surge AI was founded by Edwin Chen, a former Google and Meta engineer, in 2020. The company generated over $1 billion in revenue last year, surpassing Scale AI which reported $870 million.

After Meta invested $14.3 billion in Scale AI—for a 49% stake in the company and its CEO Alexandr Wang to lead its Superintelligence lab—, the company was valued at around $29 billion.

Now, Surge AI is leveraging recent developments to offer and expand its services to tech giants as companies such as OpenAI and Google express concerns that Scale AI could expose sensitive information to Meta. The startup has quietly positioned itself as one of the most valuable players in the data labelling industry.

SurgeAI has gained a strong reputation in Silicon Valley for its “premium, high-end data labeling services,” as reported by Reuters.

The company has capitalized on reinforcement learning from human feedback (RLHF), a technique used for training advanced AI models. Some investors see the data labeling sector growing and in need, while others hesitate as the need for human labor decreases. The success of the deal would depend on how investors perceive Surge AI’s long-term prospects.

Besides tech companies, Surge AI’s competitor, Scale AI, has also partnered with governments such as Qatar to modernize services with AI tools, and the United States to develop Thunderforge, a military planning and decision-making project.