Does a missing robots.txt hurt SEO?

A 404 for robots.txt means "no rules": crawlers fetch everything, which is fine for a small public site. What you lose is the Sitemap line, which is how Bing and AI crawlers discover the sitemap, so publish a minimal file with Allow: / and an absolute Sitemap line.

Should I block GPTBot?

That is a policy choice. Blocking GPTBot keeps the site out of OpenAI model training but does not stop ChatGPT search from citing it (that is OAI-SearchBot). AstraVerify treats blocking training-only crawlers as a recommendation, not a failure, and blocking answer-engine or search crawlers as important or critical.

Can robots.txt remove a page that is already indexed?

No. Blocking a URL in robots.txt stops crawling, so the engine cannot see a noindex tag on it; the URL can stay in the index with no snippet. To remove a page, allow crawling and return noindex or 404/410.

Why does the Sitemap line have to be absolute?

The robots.txt standard defines Sitemap as a full URL. Google resolves relative paths as a courtesy; Bing and most AI crawlers do not, so they never find the sitemap.

robots.txt for search and AI crawlers

robots.txt is a plain-text file at the root of your site (https://example.com/robots.txt) that tells crawlers which paths they may fetch. Every well-behaved crawler reads it before anything else: Googlebot and Bingbot for search, OAI-SearchBot, PerplexityBot and ClaudeBot for AI answers, GPTBot and CCBot for model training. It is the first thing a discoverability scan looks at because one wrong line can make the whole site invisible.

The file is advisory: it does not password-protect anything and bad actors ignore it. Its job is to steer legitimate crawlers, point them at your sitemap and keep them out of paths that waste their time (search results, calendars, API endpoints).

A robots.txt that works for search and AI engines

For a public marketing or product site the file is short: allow everything, exclude machine paths, and name the sitemap with a full URL.

Example record

Name: https://example.com/robots.txt
Type: text/plain
Value: User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ Sitemap: https://example.com/sitemap.xml

Serve it with HTTP 200 and Content-Type text/plain. A robots.txt that returns HTML (a soft 404) is treated as unreadable by some crawlers and as "no rules" by others.
Keep it ASCII. Smart quotes and non-breaking spaces from a CMS editor break parsing.
The Sitemap line must be an absolute URL on the canonical host. Google tolerates a relative path; Bing and most AI crawlers ignore it, and we have watched GPTBot read robots.txt dozens of times without fetching a single page until the line was made absolute.

Which crawler tokens matter

Rules apply to the most specific User-agent group that matches the crawler’s product token; a crawler with its own group ignores the * group entirely. These are the tokens worth knowing:

Googlebot, Bingbot: classic search. Blocking them removes the site from search results and from AI Overviews and Copilot, which are built on those indexes.
OAI-SearchBot (ChatGPT search), PerplexityBot, ClaudeBot, Google-Extended (Gemini grounding): answer engines that fetch pages live to cite them. Blocking them means the site is never quoted in those answers.
GPTBot, CCBot, Applebot-Extended, anthropic-ai: training crawlers. Blocking them keeps the site out of future model training; it does not affect search or live citations.
AstraVerifyBot: this site’s scanner. Disallow it and the scan reports the rule and stops.

Mistakes that block real pages

Disallow: / under User-agent: * left over from staging. The most common cause of a site vanishing after launch.
Wildcard collisions: Disallow: /*/members meant for tenant pages also blocks /platform/members. Add explicit Allow lines and confirm in a crawl log that bots fetch the route.
A CDN "block AI bots" toggle that returns 403 at the edge while robots.txt says Allow. The crawler never sees the page and never reads your rules; only a request with that crawler’s user-agent shows it.
Bot Preference Sync features that rewrite robots.txt for you, adding training-bot blocks you did not choose.
Blocking CSS and JavaScript paths. Google renders pages; blocked assets mean a blank page in its eyes.

How to check it

Fetch https://yourdomain/robots.txt in a browser and read it. Then run the domain through AstraVerify: the Crawl access card shows which crawler tokens are allowed on the root path, whether the Sitemap line is absolute, and, separately, what each crawler’s user-agent actually receives from your edge. After a change, press Verify on that card to re-check only those requests.

Frequently asked questions

Does a missing robots.txt hurt SEO?: A 404 for robots.txt means "no rules": crawlers fetch everything, which is fine for a small public site. What you lose is the Sitemap line, which is how Bing and AI crawlers discover the sitemap, so publish a minimal file with Allow: / and an absolute Sitemap line.
Should I block GPTBot?: That is a policy choice. Blocking GPTBot keeps the site out of OpenAI model training but does not stop ChatGPT search from citing it (that is OAI-SearchBot). AstraVerify treats blocking training-only crawlers as a recommendation, not a failure, and blocking answer-engine or search crawlers as important or critical.
Can robots.txt remove a page that is already indexed?: No. Blocking a URL in robots.txt stops crawling, so the engine cannot see a noindex tag on it; the URL can stay in the index with no snippet. To remove a page, allow crawling and return noindex or 404/410.
Why does the Sitemap line have to be absolute?: The robots.txt standard defines Sitemap as a full URL. Google resolves relative paths as a courtesy; Bing and most AI crawlers do not, so they never find the sitemap.

Check your own domain. The scan shows your live records, a score out of 100 and the exact record to publish for each fix.