ChatGPT search found hundreds of millions of new pages through a Cloudflare pilot

On 15 October, Cloudflare and OpenAI said a research pilot using Cloudflare's network signals had led OpenAI to hundreds of millions of new pages, which it indexed for ChatGPT search. About 1% of Cloudflare's network took part. It affects any South African business that wants to be found in ChatGPT, especially sites running through Cloudflare.

What changed

Cloudflare announced the pilot in July, and the first results came out on 15 October in a post on its blog co-written by two OpenAI staff members. Cloudflare can see things a crawler can't: when a page last changed, what changed on it, and the quality of a site's traffic. OpenAI's search crawler uses those hints to choose which pages to fetch. The post's examples are everyday ones, like a washing machine's user manual, current passport fees and requirements, or a restaurant's menu and opening hours. Pages like that, the companies argue, are useful even when hardly anyone links to them.

The same signals are meant to cut wasted crawling. Bots keep requesting files that aren't on the server, some sites bounce almost any address to the home page, and others serve the same page under more than one address. Change signals tell the crawler whether a return visit is worth it. The companies are also testing a way to flag one harmful page, such as a bogus sign-in form slipped onto an otherwise sound site by an attacker, and keep it out of the index without the whole domain paying for it.

The post is thin on proof. There are no figures for ChatGPT citations of the newly found pages, how many visits they sent or how much server load dropped, and no word on which sites took part. It even admits that finding new pages can push crawling up for a while as they're assessed. Cloudflare has said it plans to open the capability more widely before the end of 2026. The October post gives no date.

The October post puts the limit plainly: "A signal about a page doesn't grant permission to access it." Your own crawler settings still decide what OpenAI may fetch.

What it means for South African businesses

For South African businesses, the telling detail is the kind of page the companies say is worth finding. A Durban plumber's call-out fees, a Stellenbosch restaurant's winter trading hours, the medical aids a Pretoria practice accepts: plain pages that rarely earn a backlink. If ChatGPT search gets better at finding pages like these, a small local business with few inbound links has a better chance of being read. Being found is only step one, though. OpenAI ranks what it indexes by its own rules, and the pilot promises no citations.

The announcement doesn't name South Africa or any other country, and the companies haven't said which sites were in the 1% cohort, so you can't tell from outside whether any South African sites took part. What you can check is whether OpenAI's crawlers reach your site at all. Cloudflare's AI Crawl Control, which the October post points readers to and which comes with every Cloudflare plan, lists the AI services visiting your site and lets you allow or block each one.

OpenAI keeps search and training apart, with a separate robots.txt rule for each, as OpenAI's crawler page sets out. OAI-SearchBot collects pages for ChatGPT search; block it and your pages drop out of ChatGPT's search answers, apart from bare navigational links. GPTBot gathers content that may train OpenAI's models. Allowing the first and blocking the second keeps you in ChatGPT search while asking OpenAI not to train on your pages. Allow both and OpenAI may use one crawl for both.

Tracking is the weak spot. Analytics tools spot ChatGPT visits by their referrer or UTM tag, and Search Engine Journal reported that some ChatGPT links carried neither earlier in October. If your ChatGPT traffic dipped then, suspect the tracking before your content. Getting access and tracking right comes first in any AI search optimisation plan, and you can do most of it this week.

What to do now

1. Log in to Cloudflare, open AI Crawl Control and check that OAI-SearchBot is allowed. If your site isn't on Cloudflare, ask your host or developer whether a firewall rule or security plugin blocks AI crawlers.
2. Open yourdomain.co.za/robots.txt in a browser and make sure no rule blocks OAI-SearchBot. Then decide on GPTBot as a separate question. OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search.
3. List the pages a customer is most likely to ask ChatGPT about: prices, trading hours, service areas, booking rules, product manuals. Give each its own address, put the answer in plain text near the top and keep it current, since how recently a page changed is one of the signals Cloudflare named. Fix pages that redirect to the home page or duplicate another.
4. Ask whoever runs your hosting to search the server logs for OAI-SearchBot. OpenAI publishes the IP ranges its crawlers use, so you can confirm visits are real and check your firewall lets those ranges in.
5. In your analytics, save a report of visits from ChatGPT by referrer and UTM tag, and mark the tracking gap from earlier in October on it so nobody reads it as lost interest.

Sources

• ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot, Search Engine Journal, 20 October 2026
• Overview of OpenAI Crawlers, OpenAI, 21 October 2026