Letting AI crawlers into your website is one thing. Being understood once they’re there is another. An AI assistant deciding who to recommend doesn’t browse your site the way a customer does, it grabs the raw content, checks it against what the buyer asked for and drops anything it can’t confirm in seconds.
That means a website can be perfectly crawlable and still be passed over because the opening hours load via a script, the prices sit inside an image or the business address differs from one directory to the next. If you haven’t yet checked whether AI crawlers can get in at all, start with our guide to AI crawlers, robots.txt and Cloudflare.
Put key copy in the HTML (not behind JavaScript, images or tabs)
Your most important information such as what you do, where you work, who you work with, prices, opening hours and contact details, should be in the HTML your server sends, not loaded after scripts run.
This matters because most AI crawlers don’t run JavaScript. A 2024 study by Vercel and MERJ found that none of the major AI crawlers it tracked, including those from OpenAI and Anthropic, rendered it. Googlebot does, which is why a site can look fine in Google and almost blank to ChatGPT.
Metehan Yesilyurt, a GEO researcher at Peec AI, saw this clearly in a recent experiment: A site of 50,000 statistics pages that Google largely ignored but ChatGPT fetched around 72,000 times in 30 days. His conclusion is that “ranking on Google and being retrievable by an LLM are two different problems”, and his advice is to “use server-rendered HTML instead of hiding data behind JavaScript, tabs, or images” (Peec AI).
One important caveat: his statistics were deliberately invented as research, and they ended up cited in an academic paper and bank research. The lesson is about how AI chooses sources, not a tactic to copy (only ever publish facts you can stand behind).
In practice, that means:
- Server-side rendering or static HTML for key pages
- No prices, opening hours or contact details locked inside images
- Crucial answers visible on the page, not hidden in accordions or tabs that only load on click
To check your own site, right-click your homepage, choose “View page source” and search for a sentence from the page. If it isn’t there, AI crawlers probably can’t see it either.
Add structured data and keep your details consistent
When an AI agent builds a shortlist, one of its steps is verifying the facts: does this business really offer what the buyer asked for, in the right place, at the right time? Structured data (schema markup) gives it a machine-readable summary to check against.
At a minimum, most businesses should have Organisation or LocalBusiness schema covering:
- Business name, address and phone number
- Opening hours
- Services and area served
- FAQ content marked up where it fits
Just as important is consistency. Your name, address and phone number should match exactly across your website, Google Business Profile, LinkedIn and directories. Send mixed messages and AI will pick a competitor whose details agree.
Measure it
Once the technical basics are in place, track whether AI platforms are actually visiting and citing you:
- AI bot hits in your server logs or Cloudflare. Are OAI-SearchBot, Claude-SearchBot and PerplexityBot visiting – and getting 200 responses?
- AI referral sessions in GA4. Set up a channel group or segment for referrals from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com.
- Citations across a fixed set of prompts. Pick ten questions your buyers actually ask – “best accountant in Liverpool for a small limited company”, for example – and check monthly which brands each assistant names.
- New referring domains and unlinked mentions. Use Ahrefs or Semrush to spot new links, and search for your brand name and key facts in quotes to find mentions.
Tracking the same prompts each month is the most useful of these. It shows not just whether you appear, but how your share of AI answers compares with competitors.
Frequently asked questions
Do AI crawlers run JavaScript?
Most don’t. Googlebot renders JavaScript, but research from Vercel and MERJ found the major AI crawlers, including those from OpenAI and Anthropic, did not. Keep key content in server-rendered HTML.
What is llms.txt and do I need one?
llms.txt is a proposed standard for a plain-text file that summarises your site for AI tools. It’s cheap to add, but it isn’t a substitute for the basics – allowing the right crawlers and serving readable HTML matter far more.
Readable is the start, not the finish
Once AI can get in and understand your site, being recommended depends on what it finds. AI rewards businesses that are clear (one consistent description of who you are, what you do and who it’s for), backed up (reviews, directories, press and partner pages saying it too) and easy to quote (plain answers to real customer questions). We’ll cover each of those in a future guide.
In the meantime, find out how your site scores today. Our free AI Readiness Check tests your homepage against more than 20 AI visibility signals across ChatGPT, Gemini, Claude and Perplexity in around 30 seconds, no sign-up needed. New to this series? Start with Can AI read your website?