Your website can’t be recommended by AI if the doors are shut. And on a surprising number of business websites, they are. Not because anyone decided to block ChatGPT or Perplexity, but because of a robots.txt file nobody has looked at in years, a security setting left on its default or a Cloudflare update that changed the rules overnight.
If you’ve run the three checks in our previous guide and something looked wrong, this is the article to forward to whoever runs your website. Each point explains what to check and why. Most fixes take minutes, and getting them right can be the difference between being on an AI’s shortlist and being invisible to it.
Allow the AI search crawlers in robots.txt
Your robots.txt file (at yourdomain.co.uk/robots.txt) tells crawlers which parts of your site they can visit. Each AI platform uses several crawlers, and the ones that matter for visibility are the search and user crawlers – the ones that index your pages and fetch them when someone asks a question.
| Crawler | Run by | What it does |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches pages when a ChatGPT user asks a question |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude’s search results |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question |
| PerplexityBot | Perplexity | Indexes pages to surface and link them in Perplexity answers |
| Perplexity-User | Perplexity | Fetches pages in response to a user’s question |
| Bingbot | Microsoft | Crawls for Bing, whose index feeds Microsoft Copilot |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode |
OpenAI is clear about the stakes: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers” (OpenAI). It also notes that robots.txt changes can take around 24 hours to take effect. Anthropic says blocking Claude-SearchBot “prevents our system from indexing your content for search optimization” (Claude support).
A simple robots.txt that welcomes the search crawlers:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bingbot
User-agent: Googlebot
Allow: /
Sitemap: https://www.yourdomain.co.uk/sitemap.xml
One nuance: User-triggered fetchers don’t always obey robots.txt. OpenAI says that for ChatGPT-User “robots.txt rules may not apply”, and Perplexity says Perplexity-User “generally ignores robots.txt rules” (Perplexity). Robots.txt still controls the indexing crawlers, but it means your firewall settings matter just as much.
Training bots: a business decision, not a visibility one
Some crawlers collect content to train future AI models rather than answer questions today: mainly GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Google).
Blocking GPTBot or ClaudeBot doesn’t stop you being found in ChatGPT or Claude search answers; those rely on the search crawlers above. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central), but it also covers grounding in Gemini apps, so blocking it may reduce how often Gemini draws on your content. For most small businesses that want to be recommended, we’d lean towards allowing them (but make it a deliberate choice).
Check Cloudflare’s AI crawler settings
If your site uses Cloudflare, check its AI settings now. Since July 2025, Cloudflare has blocked AI training crawlers by default on new domains unless the owner chooses otherwise.
It went further on 15 September 2026. Cloudflare now sorts AI traffic into three categories – Search, Agent and Training – and for new domains it blocks Training and Agent traffic by default on pages that display ads. The Agent category includes user fetchers such as ChatGPT-User. Crucially, multi-purpose crawlers like Googlebot and Bingbot are now judged by their most restrictive behaviour, so a site that blocks Training could end up blocking Googlebot and Bingbot too. Review the AI settings under Security in the Cloudflare dashboard and set each category deliberately.
Firewalls and bot-fight modes
Even with robots.txt and Cloudflare’s AI settings correct, security tools can still turn crawlers away. Cloudflare’s Bot Fight Mode, Wordfence and other web application firewalls or a hosting provider’s own bot protection can challenge or block anything that looks automated. Check the firewall logs for blocked requests from the crawlers in the table above and allow-list them where needed, OpenAI, Anthropic and Perplexity all publish details to help verify their crawlers are genuine. For the wider picture, see our guide on how web hosting affects SEO.
Test it
Don’t take anyone’s word for it. Fetch key pages using each crawler’s user agent and confirm a 200 (OK) response containing real text – not a block page, a CAPTCHA or an empty shell:
curl -A “OAI-SearchBot” -s -o /dev/null -w “%{http_code}” https://www.yourdomain.co.uk/
Repeat for each crawler and for your most important pages: homepage, service pages, location pages and contact page. This only tests rules based on user agent – some firewalls also check where a request comes from – so cross-check against your server or Cloudflare logs.
Does blocking GPTBot stop ChatGPT citing my website?
No. GPTBot is OpenAI’s training crawler. ChatGPT search relies on OAI-SearchBot, and user-requested fetches come from ChatGPT-User. As long as those are allowed, blocking GPTBot shouldn’t stop you appearing in ChatGPT search answers.
Open the doors, then check they’re open
Getting AI crawlers through the door is the first technical step to being recommended. Once they’re in, they need to be able to read and verify what they find – we cover that next in Can AI understand your website?
Not sure where your site stands? Our free AI Readiness Check tests your homepage against more than 20 AI visibility signals in around 30 seconds. And if it flags problems you’d rather not tackle yourself, get in touch (we’ll fix them for you).