If you’ve read any tech news in the last week, you may have seen headlines saying that websites can soon “block AI” with a single line of code. That’s partly true. But “block AI” covers several quite different things, and if you run a small local business, blocking the wrong one could make you harder for customers to find.
So here’s what’s actually been announced, what it does and doesn’t do, and what I think most small businesses should do about it (spoiler: probably nothing dramatic).
On 30 October, Google announced that it plans to support “a shared AI training control in robots.txt”, which would let websites “express their AI training preferences across multiple AI providers in a single step.” Microsoft announced the same for Bing on the same day, and Search Engine Journal reports that OpenAI (the company behind ChatGPT) has made a similar commitment. The proposal itself was written by someone at Anthropic, which makes the Claude AI assistant, and it’s being discussed by the Internet Engineering Task Force (the IETF, a group that develops internet standards).
That’s a fair number of the big names agreeing on something, which doesn’t happen often.
First, what’s robots.txt?
Most websites have a small text file called robots.txt. It’s a list of polite instructions for the automated programs (often called bots or crawlers) that visit websites to read them. Search engines use crawlers to find and catalogue your pages, and AI companies use them to gather material.
You can’t see robots.txt on your website unless you go looking for it, and most small business owners have never touched theirs, which is usually fine.
At the moment, if you want to stop AI companies training on your content, you have to name each company’s crawler separately, and there are a lot of them. The new idea is a single line that every participating company agrees to respect.
The important bit: training is not the same as being found
This is where the headlines get muddled, so it’s worth taking slowly. There are roughly three ways an AI or search company might use your website.
Training. This is where your content is used to teach an AI model how to write and what things are. It’s absorbed into the model in a general way, a bit like a student reading a lot of books. The model doesn’t usually quote you or link to you as a result.
Grounding. This is where an AI looks up your website at the moment someone asks a question, so it can give an up-to-date answer, often with a link back to you. If someone asks Google’s AI Mode or Bing’s Copilot “who does same-day boiler repairs in Rugby?”, grounding is what lets it find and mention your business.
Normal search. The good old list of blue links, plus maps and Business Profiles.
The new shared control is about the first one only: training. Google says that if a website uses it, Google will treat it as though the site had blocked Google-Extended (Google’s existing AI training control). Google’s own documentation says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” Microsoft says the same for Bing rankings.
Appearing in Google’s AI Overviews and AI Mode is controlled separately, through a setting in Google Search Console (Google’s free tool for website owners). Google is clear that sites which opt out of that “will not receive traffic or impressions from our generative AI features.”
Bing is also introducing separate grounding controls, and it warns that websites which switch grounding off “will see reduced brand visibility in AI experiences”, including Copilot. OpenAI separates things too: its documentation says a website can allow its search crawler “in order to appear in search results while disallowing GPTBot”, its training crawler.
When does this start?
Not yet. Google says it aims to support the new control “in 2027”. Bing’s new settings start taking effect from 1 February 2027. OpenAI is reported to be planning support in the coming months, and Anthropic says it has started work on it. The proposal is still a draft, and people involved in the discussions say there are still areas of disagreement, so the details could change.
It’s also worth knowing that robots.txt is a request, not a lock. Reputable companies follow it, but there’s no law in the UK that forces AI companies to, and the government said in March that a broad copyright exception with an opt-out is no longer its preferred way forward.
So, should you use it?
For most local service businesses, I don’t think blocking AI training makes much difference either way, and I wouldn’t lose sleep over it. Your website is there to tell people who you are, what you do, where you work and how to get in touch. If an AI model has learned in a general sense that “Smith & Sons are plumbers in Rugby”, I struggle to see the harm. If anything, it’s mildly useful.
What I would be very careful about is blocking grounding or opting out of AI search features. That’s the part that can actually send you customers. More and more people are asking AI assistants for recommendations, and if your site has told them to stay away, you simply won’t be in the answer. Your competitor will.
There are exceptions. If your content is the thing you sell, the calculation changes. A photographer whose portfolio is their livelihood, a food blogger with original recipes, a trainer who sells courses, or a writer who puts real effort into in-depth guides may reasonably decide they don’t want that work used to train a model that could then compete with them. In those cases, the new training control looks like a sensible, low-risk option once it’s supported, because it shouldn’t affect your search visibility.
The other thing I’d say is: don’t let anyone talk you into blocking everything “just to be safe”. I’ve seen robots.txt files that accidentally block Google altogether, and the result is a website that quietly vanishes from search. A one-line mistake in that file can do far more damage than any AI company.
What I’d do now
Nothing urgent. None of the companies supports the new control yet, so there’s no rush.
If you’re curious, you can see your own robots.txt by typing your web address followed by /robots.txt into your browser (for example, yourwebsite.co.uk/robots.txt). If it contains things you don’t understand, or someone added AI blocks a while ago, it’s worth having it checked.
If you use Google Search Console, check that you haven’t switched off AI features without meaning to. And if your website platform or SEO plugin offers a “block AI bots” button, find out exactly what it blocks before you press it.
If your content is your product and you want to protect it, make a note to revisit this in early 2027, when Google and Bing are due to start honouring the new control.
And if you’d like to understand how AI search actually picks businesses to mention, my article on preparing for AI search covers the basics. The short version is that good, honest SEO still does most of the heavy lifting.
Need a hand?
If you’d like me to check your website’s robots.txt file, or you’re not sure whether something on your site is blocking search engines or AI tools, I’m happy to take a look and explain it in plain English. Just get in touch.
