Google says robots.txt can't control what AI does with pages it has crawled

In a Search Central video released on Wednesday, 21 October 2026, Google said robots.txt decides which pages a crawler may fetch and nothing more. It can't set terms for how AI systems use what they've already collected. That matters to any South African business that wants Google Search traffic but not AI training on its pages.

What changed

The video is the second in a new series where Google's search relations team answers edge-case questions from site owners. This one asked whether a site can let AI crawlers in but keep its content out of model training. Under RFC 9309, the standard behind the file, robots.txt only states access preferences. In Google's words, it "doesn't dictate what a system does with the data after it has already crawled it." To keep a particular AI crawler off your site, you name its user-agent token in the file.

Inventing your own rule gets you nowhere. The episode showed a made-up anti-training line and explained that a parser throws it away. Google's robots.txt documentation lists the four fields it reads: user-agent, allow, disallow and sitemap. Anything else, crawl-delay included, is skipped without a warning. Other crawlers may treat unknown lines their own way, which leaves you guessing.

Google's real lever is a token called Google-Extended. No crawler sits behind it. Google fetches pages with its usual user agents and reads the token as a control. Its list of common crawlers says the token decides whether crawled content can train future Gemini models and ground answers in Gemini Apps and in Vertex AI's grounding with Google Search. The same page says it has no effect on whether a site appears in Search and plays no part in ranking. Nor does it take pages out of AI Overviews or AI Mode: Google's page on AI features and your website says those are part of Search, which Googlebot's rules control.

A proper usage rule could arrive later. An IETF draft dated 19 August, co-written by one of the video's hosts, would add a Content-Usage field to robots.txt, plus a matching HTTP header, so a site could allow crawling and still ask for train-ai=n. It hasn't gone to the IETF's steering group yet, and the current version lapses on 20 February 2027 unless someone revises it.

The last tip covered paths with accented letters or emoji. Google's parser converts raw UTF-8 into percent-encoded form and treats both as the same rule. Older crawlers might not, so Google suggests writing the encoded version.

What it means for South African businesses

Nothing here waits on a rollout. Neither the video nor Google's robots.txt documentation names South Africa or any other country, because both describe how crawlers read a file any site can publish, so a .co.za site is treated like any other.

Blocking Googlebot is the trap. It only stops Google crawling those pages from then on and, as the video said, can't reach pages already crawled. Google's robots.txt documentation adds that a blocked URL can still appear in search results with no description.

For a Pretoria physio or a Gqeberha car dealer, that's a terrible deal. Blocking Google-Extended keeps Gemini training and grounding off your pages and leaves Search alone. Other AI companies publish their own tokens, and OpenAI splits search from training, as our post on ChatGPT search crawlers explains.

Businesses whose writing is the product have the most to weigh. Law firms that publish plain guides to evictions or deceased estates might want Gemini to stop training on that work. A plumber in Boksburg wants the opposite: to be read and quoted by every AI tool going. Our view is that each AI line in your robots.txt should be a choice you made on purpose, crawler by crawler, with a comment in the file saying why.

Encoding matters mostly for bilingual sites. If your Afrikaans pages keep an ê or ë in their addresses and you block any of them, encode those paths so every crawler reads the rule the same way. An AI crawler access check is part of the first month of every AI search optimisation plan we run.

What to do now

1. Open yourdomain.co.za/robots.txt in a browser and look for home-made no-AI lines. Google ignores them, so they don't stop Gemini training. If you want it off, add a Google-Extended user-agent group instead, as in the next step. Leave the rest alone: Google skips crawl-delay, but other crawlers may use it.
2. To keep your pages out of Gemini training and grounding while staying in Search, add a group with user-agent: Google-Extended on one line and disallow: / on the next. Leave the Googlebot group alone, and add a # comment with the date and the reason.
3. For every other AI company you care about, look up the token it publishes and give it its own group, deciding search and training separately where the company splits them. A bot that ignores robots.txt altogether needs a firewall rule from your host.
4. After any change, run your home page and two or three key service pages through URL Inspection in Search Console and confirm crawling is still allowed. If pages show as blocked when they shouldn't be, the technical audit in our SEO packages finds the cause.
5. Put 20 February 2027 in your calendar. That's when the Content-Usage draft lapses unless it's revised, and a sensible day to check whether robots.txt has gained a real way to state AI training preferences.

Sources

• Google Explains The Limit Of Robots.txt For AI Content Use, Search Engine Journal, 22 October 2026
• How Google interprets the robots.txt specification, Google Crawling Infrastructure, 31 August 2026
• List of Google's common crawlers, Google Crawling Infrastructure, 15 October 2026
• AI features and your website, Google Search Central, 10 December 2025