Why does ChatGPT say it cannot access my site?

Almost always because the request from OAI-SearchBot or ChatGPT-User was refused by your CDN or firewall with a 403 or a challenge page, even though a browser gets the page. Fetch the page with that user-agent from outside your network to confirm, then allow the crawler in the bot settings.

Is allowing AI crawlers a security risk?

They fetch the same public pages a browser can. Keep private paths behind authentication and out of the sitemap; that protects them from every crawler regardless of robots.txt.

Does blocking GPTBot stop ChatGPT from citing me?

No. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User; GPTBot is the training crawler. Block or allow each on its own.

How quickly do changes show up?

Crawler access is immediate for the next fetch; press Verify on the Crawl access card to confirm. Appearing in answers depends on the engine re-fetching and on your pages having something quotable, typically days to weeks.

How to make your site readable by ChatGPT, Perplexity and Claude

Answer engines work in two steps. A search index (Google’s, Bing’s or their own) finds candidate pages, then a crawler fetches those pages live to read and quote them. If that fetch is refused, the engine falls back to whatever third parties say about you: a directory listing, a review site, an old press release. The single most common reason a site is missing from AI answers is not content; it is that the crawler received a 403 or a bot-challenge page.

This guide covers the three places a fetch can fail, how to see it happening, and what to change.

Where AI crawlers get blocked

At the CDN or firewall. Cloudflare’s AI Crawl Control, Bot Fight Mode and similar products in Akamai, Fastly and AWS WAF return a 403 or a JavaScript challenge to GPTBot, ClaudeBot, PerplexityBot and CCBot by default on many plans. The origin still returns 200 to a browser, so nobody notices.
In robots.txt. A group for the crawler’s token with Disallow: /, sometimes added by a plugin or a "block AI" toggle that rewrites the file.
In the page itself. Content rendered only by JavaScript after load, or a site that shows a cookie wall or interstitial to first-time visitors. Most AI crawlers do not execute JavaScript.

How to see what a crawler sees

Request the root page with the crawler’s user-agent string from outside your network and compare the status code with a normal browser request. AstraVerify’s "Fetch as crawler" does exactly this for Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, ClaudeBot and GPTBot and shows the status, the redirect chain, the response headers and the first 2,000 characters of text the crawler would extract. A 403, a 503 or a page titled "Just a moment" is a block.

For confirmation over time, read your CDN’s crawl log filtered to verified bots: pages fetched versus robots.txt and sitemap fetches. A crawler that reads robots.txt every day but never a page is being refused somewhere.

What to change

Example record

Name: robots.txt (add)
Type: text/plain
Value: User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: /

Cloudflare: AI Crawl Control → set Search, Agent and Training crawlers to Allow (or allow the specific ones you want). Security → Bots: turn off "Block AI bots"; keep Bot Fight Mode off on a marketing site and use targeted WAF rules for scanners instead. Turn off Bot Preference Sync so it does not rewrite robots.txt.
Other WAFs: add an allow rule for the crawler user-agents, verified by IP range where the product supports it, ahead of any bot-blocking rule.
robots.txt: give the answer-engine crawlers an explicit Allow: / group (see the robots.txt guide).
Serve the important text server-side. A hero line and buttons give an extractor nothing to quote; a definitional paragraph, a "what it does" list and question-form sections do.
Publish /llms.txt so the engine has a plain-text summary written by you.

Search, answer and training crawlers are different decisions

You can allow answer engines to cite you while opting out of model training, or the reverse. Search crawlers (Googlebot, Bingbot) feed the indexes everything else is built on, so blocking them is always the most damaging. Answer-engine crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) decide whether you are quoted today. Training crawlers (GPTBot, CCBot, Applebot-Extended) decide whether the next model knows you from your own words. AstraVerify scores the three tiers with decreasing severity for exactly this reason.

Frequently asked questions

Why does ChatGPT say it cannot access my site?: Almost always because the request from OAI-SearchBot or ChatGPT-User was refused by your CDN or firewall with a 403 or a challenge page, even though a browser gets the page. Fetch the page with that user-agent from outside your network to confirm, then allow the crawler in the bot settings.
Is allowing AI crawlers a security risk?: They fetch the same public pages a browser can. Keep private paths behind authentication and out of the sitemap; that protects them from every crawler regardless of robots.txt.
Does blocking GPTBot stop ChatGPT from citing me?: No. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User; GPTBot is the training crawler. Block or allow each on its own.
How quickly do changes show up?: Crawler access is immediate for the next fetch; press Verify on the Crawl access card to confirm. Appearing in answers depends on the engine re-fetching and on your pages having something quotable, typically days to weeks.

Check your own domain. The scan shows your live records, a score out of 100 and the exact record to publish for each fix.