What is Cloudflare's new AI Index?

Cloudflare is launching a private beta of AI Index for domains on its platform: a new kind of web index that gives site owners control and visibility into how AI systems access their data.

With AI Index enabled, Cloudflare will automatically build an “AI-optimized search index” for your site, and provide APIs and tools (MCP server, LLMs.txt, search API, etc.).

Site owners retain control over what’s included or excluded from that index, who can access it, and can monetize access (via “Pay per crawl” or Cloudflare’s x402 integrations).

Currently, AI systems and crawlers often treat the web as a passive resource: they crawl broadly, without necessarily coordinating with site owners. This leads to inefficiencies, lack of control, and limited monetization for content creators.

The goal is to build a more “fair and healthy ecosystem” where site owners are participants rather than passive data sources.

How it works (in practice)

1. Enabling and maintaining the index

When a site is on Cloudflare, the owner can opt into AI Index.
New or updated pages are processed in real time. Cloudflare handles all the technical details (embeddings, chunking, storage, etc.).
Owners can control which content is exposed via “AI Crawl Control.” They can also opt out entirely.

2. APIs & standards offered

MCP (Model Context Protocol) server: allows agentic applications to query the site in a structured way.
Search API: structured JSON responses for queries.
LLMs.txt / LLMs-full.txt: a machine-readable “map” of the site, to guide AI systems how to interact with its content.
Bulk data API: for transferring larger amounts of content, under the rules the site owner sets.
Pub/sub subscriptions: AI builders can subscribe to updates from a site, receiving content changes automatically instead of re-crawling.
Discoverability directives (in robots.txt or other well-known URIs) help AI systems detect what APIs are available.

3. Monetisation & access control

Through AI Crawl Control, owners see who is accessing their content and set rules or restrictions.
Pay per crawl and other integrations let owners monetize access to their indexed content.

The “Open Index” layer

While individual site indexes provide fine-grained control, Cloudflare is also building an Open Index: a bundled, opt-in layer that aggregates many sites’ AI indexes.
Features of the Open Index:
- Unified access: query across many participating sites with a single interface.
- Filtering by content quality, originality, depth, etc.
- Monetization still flows back to individual sites via the underlying “pay per crawl” mechanisms.

Impact and call to action

This model aims to shift the balance: sites decide how their content is used, and AI builders get structured, permissioned access. Cloudflare is starting with a private beta, inviting sites and AI builders to sign up to participate. If you want to enroll your website into the AI Index or access the pub/sub web feed as an AI builder, you can sign up today.