If I block GPTBot, will ChatGPT stop citing me?

No. ChatGPT search cites pages fetched by OAI-SearchBot and opens pages on request with ChatGPT-User. GPTBot only collects training data. Each token is a separate decision in robots.txt.

Does blocking AI crawlers hurt my Google ranking?

Not directly; Google Search uses Googlebot, which you should never block. Google-Extended controls Gemini and AI Overview grounding only. Blocking answer-engine crawlers does remove you from AI answers, which is where a growing share of questions are asked.

My robots.txt allows everything, but the fetch returns 403. Why?

The block is in front of your server: a CDN bot rule, a WAF managed rule or a hosting-level bot protection. robots.txt is read by well-behaved crawlers after the connection succeeds; a 403 means the connection was refused first.

Is it safe to allow these crawlers?

They request the same public pages a browser can. Anything private should be behind authentication and out of the sitemap, which protects it from every crawler regardless of robots.txt.

Is my site blocking GPTBot, ClaudeBot or PerplexityBot?

Most sites that are missing from ChatGPT, Perplexity and Claude answers are not blocking AI on purpose. A CDN plan turned on a bot block by default, a plugin added a "block AI" group to robots.txt, or a firewall rule written for scrapers caught the crawlers too. The site owner sees a normal page in a browser and assumes everyone does.

This page is the check. It takes two minutes and needs nothing installed. It also explains what each crawler token is for, so you can allow the ones that cite you and still decline the ones that only train models, if that is the policy you want.

Check 1: what robots.txt says to each crawler

Open https://yourdomain.com/robots.txt. Look for groups (User-agent lines) naming any of the tokens in the table below, and for a wildcard group (User-agent: *). A crawler follows the most specific group that names it; if no group names it, it follows the wildcard group. Disallow: / in either place is a block. A file that lists Sitemap: with a relative path, or that is served as HTML because the site returns its 404 page, is a quieter failure: several crawlers read nothing past it.

Token	Operator	What it does	What blocking it costs you
Googlebot	Google	Builds the search index used by Google Search and AI Overviews	Everything. Do not block
Bingbot	Microsoft	Builds the Bing index used by Bing, Copilot, DuckDuckGo and ChatGPT search	Search and most answer engines
OAI-SearchBot	OpenAI	Fetches pages to cite in ChatGPT search answers	Being cited by ChatGPT
ChatGPT-User	OpenAI	Live fetch when a user asks ChatGPT to read a page	ChatGPT cannot open your pages on request
GPTBot	OpenAI	Collects training data for future models	Future models learn about you from third parties only
ClaudeBot	Anthropic	Fetches pages for Claude answers and for training	Being cited by Claude
anthropic-ai	Anthropic	Older training token; still honoured	Training only
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	Being cited by Perplexity
Google-Extended	Google	Controls use of your content in Gemini and AI Overviews grounding; does not affect Search	Gemini citations and Overview grounding
Applebot-Extended	Apple	Controls training for Apple models; Applebot itself feeds Siri and Spotlight	Apple model training only
CCBot	Common Crawl	Open web archive used to train many models	Presence in most open training sets
Amazonbot	Amazon	Alexa and Amazon answer services	Alexa answers
Bytespider	ByteDance	Training and search for ByteDance products	Little; many sites block it for load reasons

Check 2: what the crawler receives when it asks for a page

robots.txt is a request; the block that matters is what the server returns. Fetch your home page while identifying as the crawler, from outside your own network, and read the status code. 200 with your page text is open. 403, 429 or a page titled "Just a moment" or "Attention required" is a block from the CDN or firewall, and the crawler sees it even when robots.txt says Allow. A 301 to the www host followed by 200 is fine; a redirect loop or a redirect to a login page is not.

AstraVerify does this on the result page: "Fetch as crawler" requests your root page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, ClaudeBot and GPTBot and shows the status, the redirect chain, the response headers and the first 2,000 characters of text the crawler would extract. If you prefer the command line, curl with the crawler user-agent string gives the same status code.

Example record

Name: Terminal
Type: command
Value: curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])" https://yourdomain.com/ | head -1

Check 3: whether there is anything to read

A 200 is not enough if the page is empty until JavaScript runs. Most AI crawlers do not execute scripts, so a single-page app that renders everything client-side gives them a title and nothing else. View the page source (not the inspector) and look for your headline and first paragraph as plain text. If they are not there, the fix is server-side rendering or prerendering of the public pages, which also speeds up the site for people.

Where the block usually is, and how to remove it

Example record

Name: robots.txt (add)
Type: text/plain
Value: User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: /

Cloudflare: AI Crawl Control blocks AI crawlers on many plans by default and returns a text/plain "blocked" page. Set Search, Agent and Training crawlers to Allow, or allow specific ones. Under Security, Bots, turn off "Block AI bots" and keep Bot Fight Mode off on a marketing site. Turn off Bot Preference Sync so it stops rewriting robots.txt.
Akamai, Fastly, AWS WAF, Vercel and Netlify firewalls: find the managed bot rule set and add an allow for the crawler user agents ahead of it. Vercel's "Bot Protection" and Netlify's firewall rules both have AI crawler toggles.
WordPress: plugins such as security suites and "block AI" tools append User-agent groups to robots.txt or return 403 from PHP. Check Settings, Reading (Discourage search engines must be off) and the plugin's bot list.
robots.txt: give the answer-engine crawlers an explicit Allow group. Keep training crawlers blocked if that is your policy; do not confuse the two.

Confirming the fix

Edge settings apply within a minute; robots.txt changes apply the next time each crawler reads the file, usually within a day. Re-run the fetch as each crawler and expect 200 with your text. Then watch the crawl log in your CDN for verified bots: a crawler that reads robots.txt daily but never fetches a page is still being refused somewhere. AstraVerify's Crawl access card has a Verify button that re-checks only this part.

Frequently asked questions

If I block GPTBot, will ChatGPT stop citing me?: No. ChatGPT search cites pages fetched by OAI-SearchBot and opens pages on request with ChatGPT-User. GPTBot only collects training data. Each token is a separate decision in robots.txt.
Does blocking AI crawlers hurt my Google ranking?: Not directly; Google Search uses Googlebot, which you should never block. Google-Extended controls Gemini and AI Overview grounding only. Blocking answer-engine crawlers does remove you from AI answers, which is where a growing share of questions are asked.
My robots.txt allows everything, but the fetch returns 403. Why?: The block is in front of your server: a CDN bot rule, a WAF managed rule or a hosting-level bot protection. robots.txt is read by well-behaved crawlers after the connection succeeds; a 403 means the connection was refused first.
Is it safe to allow these crawlers?: They request the same public pages a browser can. Anything private should be behind authentication and out of the sitemap, which protects it from every crawler regardless of robots.txt.

Check your own domain. The scan shows your live records, a score out of 100 and the exact record to publish for each fix.