Should you block AI from reading your website? The new controls explained

The short answer

For most small businesses, the right move is to stay readable by the AI tools that search the web and send people to you, and only opt out of AI training if your content itself is what you sell. A new shared robots.txt rule for opting out of AI training is coming in 2027, and it’s designed so you can say no to training without disappearing from search. Blocking everything is rarely the smart choice for a business that wants enquiries.

What changed on 30 October

Two related announcements landed on the same day.

Microsoft Bing published new preferences that separate two things that used to be lumped together: using your content to train AI models, and using it to “ground” AI answers, which means reading your page live to answer someone’s question in tools like Copilot. You can read the details on the Bing blog.

At the same time, Google said on its Search Central blog that it plans to support a shared AI training control for websites, based on a proposal at the IETF (the body that works on internet standards). The idea is one line in your robots.txt file that tells every participating AI company not to train on your site, instead of a separate rule for each crawler. As Search Engine Roundtable reported, OpenAI and Anthropic have also said they’ll support it.

Training and AI answers are not the same thing

This is the part most people miss, and it’s why “block AI” is the wrong question.

• Training is when an AI company uses pages from the web to build or improve a model. Your content becomes part of what the model has learned, usually with no link back to you.
• Grounding, or live search, is when an assistant like ChatGPT, Claude, Gemini, Perplexity or Copilot looks up pages right now to answer a question, and often cites and links the sources it used.

For a business that wants customers, the second one is the valuable one. When someone asks “who’s a good bookkeeper near Geelong?” the assistant can only recommend you if it can read your site. Blocking that is a bit like taking your number out of the phone book because you didn’t like the printer.

Training is more of a values call. Some businesses don’t mind, some do, and both are reasonable positions.

How the new shared training rule works

The proposal adds a special name to robots.txt that every participating AI company agrees to respect for training. Based on Bing’s published example, a site that wants to opt out of AI training entirely would add:

• Line 1: User-Agent: __usage_ai-training
• Line 2: Disallow: /

You can also opt out just part of a site by putting a folder path after Disallow, the same way normal robots.txt rules work. Allowing training is the default, so if you’re happy with it you don’t need to add anything.

A few details worth knowing:

• Timing. Google says it aims to support the control along with other AI providers in 2027. Bing says it will start honouring the new preferences as pages are crawled from 1 February 2027. Until then, keep using the existing crawler rules described below.
• Existing controls keep working. Google’s current training control, Google-Extended, isn’t going away. Google has said that if you use the new shared rule and don’t mention Google-Extended separately, it will treat that as opting out of Google-Extended too.
• It’s still a proposal. It’s an IETF draft, not a finished standard, so the exact details could change before it’s widely supported.

What Bing’s grounding controls mean

Bing has gone a step further and given you page-level choices about AI answers, set with a meta tag or a response header rather than robots.txt. You can allow grounding (the default), limit it to Microsoft’s own products like Copilot and Bing’s AI answers, or turn it off.

Bing is clear about the trade-off. It says these choices don’t change eligibility for normal Bing search results, but limiting or blocking grounding means reduced visibility in AI experiences. In other words, you can hide from AI answers, but you’ll be less visible where more people are now asking questions.

Bing is also retiring an older approach. If your site uses NoArchive or NoCache tags to keep content out of AI, those will no longer be read as a training or grounding preference after 1 March 2027, so check with whoever built your site.

The crawlers you’ll see today

Until the shared rule arrives, AI companies use their own crawler names, and many have separate ones for training and for live search. The main ones:

• OpenAI: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User when a person asks ChatGPT to visit a page.
• Anthropic: ClaudeBot for training, Claude-SearchBot for search, and Claude-User for requests made by people using Claude.
• Google: Google-Extended controls whether your content is used for Gemini models. It does not affect your Google Search rankings, and Google’s AI Overviews and AI Mode use the normal Googlebot, so blocking Google-Extended won’t remove you from them.
• Perplexity: PerplexityBot for its search answers.
• Microsoft: Bingbot powers both Bing and Copilot, which is why Bing’s new page-level preferences matter.

The split matters. You can block the training crawlers (GPTBot, ClaudeBot, Google-Extended) and still allow the search ones, which is what many businesses choose.

Should your business block AI training?

It depends on what your website is for.

You might want to opt out of training if:

• Your content is the product, such as paid courses, original research, recipes, photography or detailed guides people would otherwise pay for
• You’re a creative business and don’t want your work used to build tools that imitate it
• You’d simply prefer not to, as a matter of principle

You probably don’t need to bother if:

• Your website exists to win enquiries, bookings or sales
• Your content is mostly about your services, prices, location and team
• You want to be the business AI tools recommend, which depends on them understanding who you are

For the second group, being in training data may even help a little, since an assistant that already knows your business exists has more to go on. But the bigger win is staying open to live search. We cover that side in how to get your business mentioned by ChatGPT, Claude and Google’s AI answers.

Check what your site is doing now

Many business owners have no idea what their robots.txt says. Platforms and security services sometimes make the choice for you:

• Squarespace has a setting to exclude known AI crawlers, so check whether it’s been switched on.
• WordPress sites can have rules added by SEO or security plugins without anyone noticing.
• Shopify lets developers customise robots.txt, so a theme or app may have changed it.
• If your site sits behind Cloudflare, its own AI crawler settings can block bots regardless of what your robots.txt says.

To see your file, type your domain followed by /robots.txt into a browser, for example yoursite.com.au/robots.txt. If you see lines naming GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot with Disallow under them, someone has made a choice. Make sure it’s the one you want.

A sensible default

For most small and medium businesses, we suggest:

• Keep Googlebot, Bingbot and the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed
• Decide deliberately about training crawlers. Block them if your content is your product, otherwise leaving them allowed is fine
• Leave Bing’s grounding preference on the default unless a specific page shouldn’t be quoted
• Add the shared training rule in 2027 if you want to opt out of training in one step, once the major providers confirm they support it
• Write the decision down, so the next developer or plugin update doesn’t quietly undo it

A quick checklist

• Look at your robots.txt today
• Check your platform and Cloudflare AI settings
• Make sure you aren’t blocking Googlebot or Bingbot
• Separate training from search in your decision
• Remove old NoArchive or NoCache tags if they were only there for AI, before March 2027
• Revisit this in early 2027, when the shared rule starts to be supported

Need a hand?

If you’re not sure what your site is telling AI tools, our free website check is a good place to start on the SEO basics, and it’s worth reading what a technical SEO audit actually fixes. If you’d like someone to review your robots.txt and crawler settings and set them up properly for both search and AI, that’s part of our SEO service.