Can an AI actually read your website?

For an assistant to quote your site, three things have to work on production. The relevant bots are allowed in. The server returns the page while the buyer is still in the conversation. The facts sit in the first HTML response, so a fetcher that does not run JavaScript can read them.

Can an AI actually read your website is mostly a technical SEO question with extra bots and extra data. If Googlebot already receives crawlable, indexable HTML, you already have most of the stack. What you still enable is named access for GPTBot and ChatGPT-User, a clear path through the CDN and WAF (Cloudflare has been shipping AI-bot controls that default to deny), and the same facts as typed data (tables plus JSON-LD) so a machine can use a number as a price or a dimension instead of guessing from a sentence. Read also: getting your company recommended by AI: supplier shortlist facts - why published specs matter once a fetcher can read the page.

In this article:

• How does a page get into an AI answer?
• Which bots should be allowed, and what does each one enable?
• Does the CDN or bot-fight layer still let those bots through?
• How much of this is already technical SEO?
• How do you return a usable response in time?
• How do you put the text in the first HTML response?
• What else should you enable so a machine can use the data?
• What does JSON-LD add once the page is readable?
• How do you verify this on production?
• What do you configure first?
• How we set this up in Drupal
• Common questions

How does a page get into an AI answer?

Assistants use two retrieval paths. Access policy has to name both, because they are different agents with different jobs.

Training crawl. A crawler fetches public pages on its own schedule. That store later feeds model training. Allow GPTBot and ClaudeBot and future models can include what you published.

Live fetch. When a buyer asks a question, the assistant opens your page during that conversation and looks for the value that was asked about. Allow ChatGPT-User and OAI-SearchBot and today's answers can cite the live page.

Write both into robots.txt and into the WAF allowlist as separate rules. One blanket "bots" line treats a training crawler and a live fetcher as the same client. They are not.

Confirm identity the same way you confirm Googlebot: published IP ranges and reverse DNS. Allow those sources. User-agent strings alone are easy to copy, so the allowlist belongs on verified ranges.

Which bots should be allowed, and what does each one enable?

Put the rules on the production host, in two places that have to agree: robots.txt and the edge (Cloudflare, AWS WAF, CDN).

In robots.txt, allow the agents in the table by name. Staging often ships with Disallow: /. Copy that file to production and every listed bot will skip the site. Your browser still loads the page, because it does not read robots.txt. Check the live URL:

curl -s "https://www.example.com/robots.txt"

You want explicit Allow (or an absence of Disallow) for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and PerplexityBot, each for the paths you want quoted. A production file that names the jobs looks like this:

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

GPTBot on those paths is how future models can train on the pages. ChatGPT-User on the same paths is how a live question can fetch them. Two User-agent groups, two outcomes. Limit Allow to the public sections you want quoted if that is narrower than the whole tree.

A third control sits on the page itself: meta name="robots" and the X-Robots-Tag header. Allow index, follow on the URLs you want cited, so Google Search can keep the page eligible for snippets and for AI Overviews and AI Mode. Those Google surfaces still run on the Google index. Allowing Googlebot is how a page stays in that index. Allowing ChatGPT-User is a separate job, on a separate agent.

At the edge, add the published IP ranges for those operators to the allowlist so verified fetchers receive HTTP 200. Akamai's 2025 figures put bots at 51% of internet traffic, with AI crawlers around a fifth of all traffic. That volume is why default WAF packs group unknown clients with scrapers. Keep abuse controls for unverified traffic. Give the named, verified AI fetchers a clear path through.

Marketing edits the CMS file. Network engineering owns the WAF. Both have to allow the same agents. A Drupal robots.txt that allows ChatGPT-User only helps once the proxy in front also returns 200 for that client.

Does the CDN or bot-fight layer still let those bots through?

The public hostname is what the fetcher hits. That is the CDN, WAF and bot-fight product, then cache, then origin. Origin can allow GPTBot in Drupal while the edge still issues a challenge, a 403, or a rewritten robots.txt. Check the edge as its own layer.

Cloudflare is the case we see most often, because it now ships AI-crawler controls on every plan. The current product names and defaults are in Cloudflare's AI bot policy docs and managed robots.txt docs.

Block AI bots / AI bot policies. Since 1 July 2025, Cloudflare asks every new domain whether AI crawlers may access the site, and the conservative choice is deny. In the dashboard this lives under Security Settings as Configure AI bot policies (the older Block AI bots toggle is being retired on 15 September 2026). Cloudflare splits traffic into Search, Agent and Training. Allow Training so GPTBot and ClaudeBot can crawl public pages for future models. Allow Agent so a live chat fetcher (ChatGPT-User and similar) can open the page during a question. Allow Search so retrieval crawlers can index for answers. From 15 September 2026, new Cloudflare domains will deny Training and Agent on pages that display ads, and leave Search allowed, unless you change that.

Managed robots.txt. Turn that setting on and Cloudflare prepends its own file in front of yours. The managed block includes Disallow: / for GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot and others, plus a content signal ai-train=no. Your Drupal Allow: / for GPTBot then sits below that prepend, and many crawlers honour the first matching group. Curl the public https://www.example.com/robots.txt. If you see # BEGIN Cloudflare Managed content and User-agent: GPTBot / Disallow: /, the edge is telling training crawlers to skip the site. Switch the managed file off, or replace those groups with Allow, when training crawl is part of the policy.

Bot Fight Mode and Super Bot Fight Mode. These products issue JavaScript challenges to traffic that looks automated. Answer-time fetchers skip JavaScript, so a challenge page is not your article. Bot Fight Mode cannot be skipped with a WAF allow rule. If it is on, turn it off for this outcome, or move to Super Bot Fight Mode and skip verified AI bots there. I'm Under Attack is the same class of challenge, site-wide.

Custom WAF and bot-score rules. A rule that challenges every client with a bot score below 30 is a common default after a security hardening ticket. Verified GPTBot and ChatGPT-User should pass Cloudflare's verified-bot list. Unverified lookalikes will not. Keep the score rule for unknown scrapers. Add an explicit allow (or skip) for the named, verified AI agents so training crawls and live fetches receive 200.

Cache. The cache rarely "blocks" by itself. The WAF in front of it does. Two cache cases still matter. A cached cookie wall or empty JS shell is what every later fetch, bot or human, receives until you purge. A 403 or challenge that was cached for that URL will keep failing after you fix the rule, until purge. After you change bot policy, purge the HTML of the URLs you want quoted, then curl again.

The same pattern exists on AWS WAF Bot Control, Akamai Bot Manager and Fastly's bot products: a managed "AI / scraper" pack sitting in front of origin. Read that pack the way you read Cloudflare's AI bot policies. Allow the verified training crawlers and live fetchers you named in robots.txt.

How much of this is already technical SEO?

Almost all of it. Crawlability, indexability, HTML in the first response, sitemaps, canonicals, hreflang, speed, and schema.org that matches the visible page: that is the same work a search team already does for Google.

Google's own guidance for AI Overviews and AI Mode is explicit about this. A page that is indexed and eligible for a snippet in Google Search is eligible to appear as a supporting link in those features. There is no extra technical bar beyond Search. Google lists the same practices: allow crawling in robots.txt and at the CDN, keep important content in text, and keep structured data matched to what the visitor sees.

ChatGPT, Claude and Perplexity reuse that same readable page. They add three extra requirements on top of a healthy SEO setup.

Named agents. Allowing Googlebot does not allow GPTBot or ChatGPT-User. Each operator ships its own user agent. You enable each one for the outcome in the table above.

A live fetch with a short clock. Googlebot can come back. An answer-time fetcher is inside a chat. HTML that arrives in the first seconds is HTML that can be quoted in that turn.

Denser, typed facts. Ranking still rewards a clear page. A buyer prompt is a list of constraints: 400 mm, 12 V, IP68, food-grade, lead time under two weeks. A machine matches those values. It prefers a spec table and JSON-LD over a paragraph that says the product "covers a wide range of sizes".

So good SEO is the base layer. The extra work is access for the new bots, and data a machine can compare without inferring. 10 SEO features a modern CMS should have covers the crawlability side; this article covers the AI agents and typed data those crawlers need once they arrive.

How do you return a usable response in time?

A live fetch runs while the buyer waits. Aim for HTTP 200 and HTML in the first seconds, on the URLs you want cited.

That means:

• 200 for verified AI fetchers on public pages.
• Rate limits sized so a short burst of answer-time fetches completes. The live agents above should reach 200, not 429.
• The article body in that 200. A cookie wall or a country picker that holds the spec behind a click leaves the fetcher with the shell. Put the citable facts (dimensions, prices, limits, industry) in the HTML that arrives with the status line.
• Time to first byte in the same range you already want for technical SEO. For a live fetch, usable HTML in time is what lets the assistant quote the page.

A sitemap is how you tell crawlers which URLs belong in that set. List the public pages you want quoted, keep lastmod in step with real edits, and point robots.txt at the sitemap. Canonicals then pick one URL when several resolve to the same product. Hreflang picks the language. Those three are SEO plumbing. They also decide which document a live fetch opens when a buyer asks in German and you publish in three languages.

How do you put the text in the first HTML response?

Answer-time fetchers read the HTML they download. Most of them skip JavaScript. Specs, prices and limits therefore belong in that first document.

Server-render the pages a buyer would ask about. The HTML that arrives should already contain the product data, the range, the constraints and the industry copy.

If the public site is a JavaScript application, keep a server-rendered route (or a prerendered snapshot) for the same facts, on one canonical URL. Text in images and SEO work the same way: a dimension that exists as text in HTML is what a fetcher can quote. A PDF attached under "download the datasheet" is the same pattern. Publish the numbers on the page, then offer the file as a copy for people who want it.

A spec table in HTML is the format that travels. One attribute per row, one unit, one value. The same numbers can then feed JSON-LD. A sentence that hides a range in adjectives does neither job.

What else should you enable so a machine can use the data?

Once the bots are allowed and the HTML holds the facts, turn on the rest of the machine layer. Each item below is an enablement: do this, and a fetcher can use the page more reliably.

llms.txt is optional. It is a short map of URLs you already made readable. Publish it if those URLs exist and you can keep the file in date. It does not grant access, it does not render JavaScript, and no major vendor has made it a ranking factor. Treat it as a pointer, after the table above is in place.

The data bar sits above ordinary SEO copy. A page can rank for "industrial pump" on adjectives and still fail a buyer prompt that asks for 400 mm, 12 V and salt water. Enable typed fields so the same value appears in the table, in the paragraph, and in JSON-LD. One edit updates all three.

What does JSON-LD add once the page is readable?

JSON-LD is schema.org in a <script type="application/ld+json"> tag. It is the same structured data you already add for Google Search. Google recommends JSON-LD as the format that is easiest to maintain, and it requires the markup to describe what is visible on the page.

On Google's AI surfaces the effect is documented as part of Search: structured data that matches the page helps Google interpret the document. There is no separate "AI schema" to invent.

On a raw ChatGPT or Claude fetch the evidence is thinner. Those fetchers read HTML. JSON-LD is a typed copy of the same facts sitting next to that HTML. We have not seen a public, repeatable test that proves JSON-LD alone moves ChatGPT citations. Do it anyway, because it is the same work as SEO, and because a typed height: 400 mm removes the guesswork a sentence still leaves.

The types that earn their keep on a B2B site, once the visible page already states the facts:

• Organization with sameAs (LinkedIn, Wikipedia, company profiles), so the model can attach the page to one company.
• Product with Offer (price or price range, currency, availability), so a cost question has a value to lift.
• Service on service pages, with the area or industry you actually serve.
• FAQPage only for questions that appear as visible Q&A on that URL.
• Article with dateModified on editorial pages, so freshness is a field, not a guess.
• BreadcrumbList, so hierarchy is explicit.

Markup that restates a vague paragraph is decoration. Markup that disagrees with the visible page is worse: Google treats mismatch as a spam signal, and a live fetcher that reads both copies now holds two numbers. Generate JSON-LD from Drupal fields so an editor change updates the HTML and the script together. Hand-written JSON next to a body field drifts within a quarter.

A minimal Product fragment, fed from fields, looks like this:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Example bilge pump",
  "brand": { "@type": "Organization", "name": "Example GmbH" },
  "additionalProperty": [
    { "@type": "PropertyValue", "name": "height", "value": "380", "unitCode": "MMT" },
    { "@type": "PropertyValue", "name": "voltage", "value": "12", "unitText": "V" }
  ],
  "offers": {
    "@type": "Offer",
    "priceCurrency": "EUR",
    "price": "240",
    "availability": "https://schema.org/InStock"
  }
}

Those properties only help if 380 mm and 12 V are also in the HTML table. JSON-LD labels the data. It does not create it. The Drupal write-up of generating this from Schema.org and metadata in Drupal is the implementation path we use.

How do you verify this on production?

Use a live URL you want cited, not a local build.

Status for a live fetcher

curl -sI -A "ChatGPT-User" "https://www.example.com/your-page"

HTTP 200 means ChatGPT can fetch that URL during a question. Repeat with -A "GPTBot" to confirm the training crawler is allowed on the same path. Read X-Robots-Tag on the same response: you want the public page indexable.

CDN / Cloudflare in front of origin

curl -sI -A "GPTBot" "https://www.example.com/your-page"

Look at the status line and the headers. cf-ray means Cloudflare answered. cf-mitigated: challenge or a 403/1020 means the edge stopped the training crawler. Repeat with -A "ChatGPT-User". Then:

curl -s -A "GPTBot" "https://www.example.com/your-page" | grep -iE 'just a moment|cf-challenge|attention required|blocked'

A match is a challenge page, not your article. In the Cloudflare dashboard, open Security Analytics (Events) for that window and filter on Bot Fight Mode, Block AI bots, and custom WAF. Origin access logs that never show GPTBot while the public site is on Cloudflare usually mean the edge returned the answer and origin never saw the request.

robots.txt on production (the file the bot actually gets)

curl -s "https://www.example.com/robots.txt"

Confirm Allow (or no Disallow) for each agent in the table, on the paths that should be public. Confirm the sitemap URL is listed. If the response starts with # BEGIN Cloudflare Managed content and lists Disallow: / for GPTBot or ClaudeBot, the managed file is in front of your CMS file. Fix that at the CDN, then curl again. Compare with the file on origin if you can reach it: they have to agree.

Facts in the HTML

curl -s -A "ChatGPT-User" "https://www.example.com/your-page" | grep -i "400 mm"

Use a string that exists in the content: a dimension, a price, a certificate number. A match means the live fetcher can read that fact. Then open the same URL with JavaScript disabled and confirm a person sees the same numbers.

JSON-LD present and in agreement

curl -s -A "ChatGPT-User" "https://www.example.com/your-page" | grep -F 'application/ld+json'

Then search the same download for the same dimension or price inside the JSON. A typed value that matches the table is the point. A script tag with a different number is a bug to fix before you add more types.

Sitemap and canonical

curl -s "https://www.example.com/sitemap.xml" | grep "/your-page"
curl -s -A "ChatGPT-User" "https://www.example.com/your-page" | grep -i 'rel="canonical"'

The page you want quoted should appear in the sitemap. The canonical should point at that same public URL.

Logs

Search a week of access logs for ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. 200s on the public pages you care about means those bots are getting through. Cross-check hits against published IP ranges with whoever runs the network.

What do you configure first?

1. Access. robots.txt plus the CDN/WAF AI-bot policy, Bot Fight Mode, and allowlists for verified ranges, per agent, so training crawls and live fetches both reach the site. Keep index, follow on the public URLs. Purge HTML for those URLs after you change a rule.
2. HTML. Specs, prices and limits in the first response, as text and tables.
3. Speed. 200 fast enough for a live fetch to finish.
4. URL signals. Sitemap, self-canonical, hreflang, so the fetch opens one current document.
5. JSON-LD. Schema generated from the same fields, matching the visible page.

Access first, because that is what lets GPTBot feed future models and what lets ChatGPT-User fetch during a question. JSON-LD last among these, because it labels data that already exists. A marketing lead can run the commands above, attach the status line, the robots snippet, a JavaScript-off screenshot and the JSON-LD grep, and hand that to whoever owns the edge and the templates.

How we set this up in Drupal

A standard Drupal page is HTML on the first response. Fields, views and layout render on the server, which is the document a live fetcher can read. The modules below then label that document, list it, and tell named bots they may fetch it. If you add a JavaScript front end, keep a server-rendered route with the same facts. Headless Drupal can expose the data as HTML for people and fetchers, and as JSON:API for the application. Both can exist. The public URL that should be quoted needs the HTML.

Fields, then pages

We model specifications as fields, not as a blob of body copy. Height, voltage, price, industry and certificate each have a field. The product page, the comparison table and the JSON-LD all read those fields. An editor changes 380 mm once. Every surface follows. Views and taxonomy then build the industry and comparison pages from the same entities, so the eleventh industry page is configuration rather than a one-off article.

Rabbit Hole is the other half of that model. A spec entity that only exists to feed a table should not be a public URL. Send those canonicals to the product page so a fetcher lands on one document.

robots.txt from the CMS

Core ships a static robots.txt in the docroot. That file is the same on every site in a multisite, and a deploy can overwrite a hand edit. For a policy you can change per environment we use RobotsTxt (drupal/robotstxt). It builds the file in the UI and stores it as config.

Rename or remove the core file first. Otherwise Drupal still serves the static copy and the module never wins. Then paste the User-agent groups from earlier in this article: Allow: / for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and PerplexityBot on the public paths. Point Sitemap: at the Simple XML Sitemap index. Export the config. On staging, keep a Disallow: / (or a Config Split overlay) so training crawlers do not treat the staging host as production.

Curl the public hostname after that. Cloudflare managed robots.txt prepends its own Disallow for GPTBot in front of this file. The CMS change only counts when the edge file matches.

Metatag: canonical, robots, title

Metatag owns the page-level signals. Set defaults per bundle, with tokens, not per node by hand.

• Canonical: the absolute alias of this entity, so ?page= and /node/123 collapse onto one URL.
• Robots: index, follow on the bundles you want quoted. noindex on thank-you pages, search faceting and unpublished layouts.
• Title and description: tokens from the same fields the H1 and the opening paragraph use, so the fetch and the snippet agree.

Add the Metatag field on a bundle only where an editor must override a default. A field on every node is how 300 pages end up with empty overrides. Views that are public landing pages get Metatag Views, so those URLs have a canonical too.

JSON-LD from the same fields

Schema.org Metatag extends Metatag and prints <script type="application/ld+json"> in the head. That is the path we use on existing sites. Map tokens to schema properties per bundle: Product name from the title, additionalProperty from height and voltage, Offer price and priceCurrency from the price field, Organization sameAs from a site-wide social URL list, FAQPage only on bundles that render visible Q&A, Article dateModified from [node:changed:html_datetime].

The mapping lives in Metatag defaults. An editor never pastes JSON. When a property is missing from the module, a small custom Metatag tag in a custom module is cheaper than a script in the theme that will drift.

Schema.org Blueprints (schemadotorg) is the other direction: it builds content types and fields from schema.org types, then emits JSON-LD and a clean JSON:API. Use it on a greenfield catalogue. Do not run both mapping systems on the same bundle.

Either way, Schema.org markup should be generated from content, not pasted by hand in a WYSIWYG. A hand-written JSON-LD block is the version that disagrees with the table after the next price change.

Sitemap

Simple XML Sitemap (4.x on Drupal 10.3/11) is the generator we install. Include the bundles you want quoted, exclude the rest. It writes hreflang alternates for translated entities, can add image URLs, and can expose a sitemap index when the tree is large. lastmod should follow the entity changed time, not a nightly rebuild of every URL.

Do not list noindex bundles. A sitemap that advertises URLs Metatag tells robots to skip is a mixed signal. After a publish, confirm the public /sitemap.xml contains the alias, then confirm Cloudflare is not serving a cached empty index.

XML Sitemap (the older project) still exists. On a new build, Simple XML Sitemap is the one with current multilingual and IndexNow support.

Aliases, redirects and breadcrumbs

Pathauto gives each bundle a stable pattern (/products/[node:title], /industries/[term:name]). Redirect then 301s /node/123 and old aliases onto that pattern, and records alias changes so a quoted URL keeps resolving.

Easy Breadcrumb can emit BreadcrumbList JSON-LD from the same trail the visitor sees. Turn that on so the hierarchy in the script matches the hierarchy in the header. A second, hand-built breadcrumb in schema_metatag on the same page is a duplicate to avoid.

Languages

Core content translation plus language prefixes already produces hreflang on many builds. A multilingual catalogue still needs one fact per language: translated fields, not a machine dump of the source. Simple XML Sitemap's hreflang sitemap is how crawlers discover the set. Canonical on a translation points at that translation, not at the source language.

Cache tags and the CDN

Drupal cache tags already know which page changed. Purge plus a purger (Cloudflare, Varnish, Fastly, or generic HTTP) drops that object at the edge. The next AI fetch receives the new HTML and the new JSON-LD. The rest of the cache stays.

Align three things after every bot-policy change: Drupal robots.txt, Cloudflare AI bot policies / managed robots file, and a purge of the HTML you just opened with curl. Origin-only tests miss the prepended Disallow.

Optional: llms.txt

llms.txt exposes /llms.txt from Drupal config and content, with cache tags so a publish refreshes the file. Generate it from the same canonical URLs Simple XML Sitemap already lists. It is a pointer, not a substitute for Allow rules or HTML.

That is the stack that lets GPTBot, ChatGPT-User and the rest of the table reach pages that already hold the facts in HTML and label those facts in JSON-LD.

Common questions

Do AI crawlers run JavaScript?

Answer-time fetchers usually read the downloaded HTML and stop. Put the facts in that first response so they can use them.

Which OpenAI bots should be allowed?

GPTBot so future models can train on your public pages. ChatGPT-User so ChatGPT can fetch during a live question. OAI-SearchBot so OpenAI search can fetch for an answer. Three agents, three jobs. Allow each one for the outcome you want.

Does Cloudflare block AI crawlers by default?

On new domains, since 1 July 2025, Cloudflare asks you to choose and the conservative option is deny. A managed robots.txt can also prepend Disallow for GPTBot and ClaudeBot in front of your CMS file. Bot Fight Mode can challenge the same clients. Curl the public hostname with -A "GPTBot" and -A "ChatGPT-User", then read Security Settings → Configure AI bot policies. Allow Training, Agent and Search for the outcomes in the table above. From 15 September 2026, new domains will deny Training and Agent on pages with ads unless you change that.

Does allowing GPTBot put me in today's ChatGPT answers?

That is the training path. Live answers use ChatGPT-User. Allow both if you want both outcomes.

Is good SEO enough for ChatGPT to read the site?

It is most of the work. A site Googlebot can crawl, with text in HTML, a sitemap and matching schema, is already readable. You still allow the named AI agents (Googlebot allow is a different rule), you still give verified ranges a 200 at the WAF, and you still publish specs as values a prompt can match.

Does JSON-LD make ChatGPT recommend me?

JSON-LD types the facts that are already on the page. On Google it is part of Search, including AI Overviews. On a raw ChatGPT fetch it is unproven as a citation lever by itself. Generate it from the same fields as the HTML so the two copies agree. That is the useful version.

Does response time affect citations?

For a live fetch, yes. HTML that arrives quickly is HTML the assistant can quote while the buyer is still in the chat.

Want this checked on your stack?

Send a production URL. We will confirm which bots are allowed at origin and at the CDN, whether Bot Fight Mode or a managed robots.txt is in the way, whether the first HTML holds the facts, whether JSON-LD matches those facts, and whether the live fetch returns in time. We do this on Drupal: robots and edge rules, server-rendered pages, fields that feed tables and schema, cache tags behind a CDN. Take a look at our content management solutions service page, or write and we will run the fetches with you.