Cloudflare and OpenAI shared early results from their AI web-discovery pilot on October 15, 2026. The companies say signals from Cloudflare's network helped OpenAI discover and index hundreds of millions of previously unknown pages for ChatGPT Search. The result is a discovery improvement, not a promise that every site behind Cloudflare will be indexed, cited or receive referral traffic.
What the pilot says it changed
Cloudflare and OpenAI launched the research pilot in July 2026. The October update says the work used a participating cohort equal to about 1% of Cloudflare's total network. Cloudflare's network signals helped OpenAI find URLs that its crawler might not otherwise have discovered efficiently. OpenAI still evaluates and ranks pages independently for how they appear in ChatGPT Search.
That distinction separates three steps that are easy to mix together: discovering a URL, indexing or processing the page, and deciding whether it is useful for a particular answer. A signal that helps a crawler find a page does not determine whether the page deserves a citation or whether a user will click through.
Why page quality and URL hygiene still matter
Cloudflare says the pilot also helped identify wasteful crawling patterns. Automated requests can make a nonexistent URL look interesting, sites can redirect many unrelated paths to the homepage, and duplicate addresses can lead to repeated copies of the same content. Filtering these cases helps a crawler spend more effort on pages that can answer a real question.
The post also describes testing signals for potentially harmful pages. A legitimate domain can contain a single compromised URL, such as a fake sign-in form. Flagging a specific page can be more precise than treating the entire domain as unsafe. These are signals for discovery and safety, not a substitute for the site owner's own security controls.
What publishers should check on their own sites
Start with pages that offer useful, specific information: product documentation, troubleshooting steps, original research, local service details and pages that answer a narrow question better than a generic overview. Then check whether those pages are technically discoverable and accessible to the crawlers you choose to allow.
- Return the right status. A real page should return a successful response. Nonexistent URLs should not return a generic homepage with a success status, because that can look like a valid page.
- Keep one preferred URL. Use a consistent canonical, avoid unnecessary parameter variants and make sure redirects lead to the intended destination rather than a broad homepage.
- Maintain useful discovery paths. Keep the XML sitemap current and use internal links that make important pages reachable. A sitemap is a discovery aid, not a guarantee of indexing.
- Update information honestly. Dates, prices, availability and instructions should reflect the real state of the page. Do not change dates only to make content appear fresh.
- Check access rules. Review robots directives and server responses so the pages you intend to expose are not accidentally blocked by another layer.
Our AI crawler controls audit workflow provides a more detailed way to review crawler access. The AI search impression workflow explains how to separate visible discovery signals from measurable traffic and search performance.
Use AI crawler controls as a policy choice
Cloudflare's AI Crawl Control lets site owners monitor AI crawler activity and set allow or block rules for individual crawlers. Use that control to make an explicit choice about access rather than assuming that allowing one crawler automatically gives every AI service permission to fetch the site. Cloudflare notes that a discovery signal does not grant permission to access a page.
The pilot announcement says it uses metadata about the web for search and discovery rather than sharing page content for model training. Keep that statement in its stated scope: it describes this research effort, not every product or data flow operated by either company. Publishers should still read the terms and controls for each service they use.
What the announcement does not promise
The published result does not provide a guaranteed indexing rate for an individual domain, a timeline for when a page will appear, a ranking formula, or a referral forecast. The pilot also does not mean a website should block or allow all AI crawlers by default. Those choices depend on the value of the content, the owner's access policy, server capacity and the kind of referrals or uses the publisher wants.
For search teams, the useful next step is to monitor server logs, crawl errors, canonical consistency and actual visits from AI services. If a page is discoverable but not useful, improve the page itself. If a useful page cannot be fetched, fix the technical barrier first. Neither problem can be solved by treating discovery as a ranking shortcut.
Sources
- Cloudflare Blog: Cloudflare and OpenAI — making AI discovery better for everyone.
- Cloudflare Docs: AI Crawl Control.