The state of email authentication and AI crawlability
Measured 2026-10-15
This is what we actually see. AstraVerify scans a domain's email authentication and its crawlability, and these are the aggregate results across every domain scanned so far: 238 domains for the email figures and 177 for the web ones, measured 2026-10-15. Counted once per domain, from its first scan, so a domain checked repeatedly does not weight the sample.
How many domains actually enforce DMARC?
69.7% publish a DMARC record. Only 26.5% set it to reject, and 45.8% reach quarantine or reject. The gap between publishing a policy and enforcing one is the single largest finding here: most domains with DMARC are running p=none, which asks for reports and blocks nothing.
85.3% publish SPF, but only 34.9% end it in -all. A record ending in ~all tells receivers to accept mail that failed the check and mark it, which is a softer position than most owners assume they are taking.
| Check | Share of domains |
|---|---|
| Can receive mail (MX record published) | 87.8% |
| Publishes SPF | 85.3% |
| Publishes DMARC | 69.7% |
| A DKIM key we could find | 66.8% |
| DMARC set to quarantine or reject | 45.8% |
| SPF ends in -all rather than ~all | 34.9% |
| DMARC set to reject | 26.5% |
What does the open web look like to an AI crawler?
94.9% allow AI crawlers in robots.txt, and 15.3% block at least one of them at the CDN instead. Those two are measured separately on purpose: a robots.txt that reads as open means nothing if the edge returns 403 to the same crawler, and only fetching the page as that crawler shows the difference.
30.5% publish llms.txt. 52.5% publish any JSON-LD at all and 42.4% publish Organization schema, which is what answer engines read to decide who a site belongs to.
| Check | Share of domains |
|---|---|
| Serves HTTPS | 97.2% |
| Allows AI crawlers in robots.txt | 94.9% |
| Root page reachable | 92.1% |
| Publishes robots.txt | 80.8% |
| Redirects to a single canonical host | 76.3% |
| Declares a canonical URL | 70.6% |
| Has a valid sitemap | 68.4% |
| Declares that sitemap in robots.txt | 63.8% |
| Sets an og:image for link previews | 56.5% |
| Publishes any JSON-LD structured data | 52.5% |
| Publishes Organization schema | 42.4% |
| Publishes llms.txt | 30.5% |
| Blocks an AI crawler at the CDN | 15.3% |
| Blocks a search crawler at the CDN | 10.2% |
How is this measured?
Each domain is counted once, from its first scan, so repeat checks do not weight the sample. The email figures read published DNS records. The web figures come from fetching the site, including fetching the root page once as each of six named crawlers.
The sample is self-selected: these are domains somebody chose to check, which skews toward domains whose owners already suspected a problem. Treat it as a picture of domains people worry about, not of the web as a whole.
Alongside the scans we have parsed 332 real DMARC aggregate reports covering 72,544 messages, sent by receivers to the domains that monitor with us.
Can I check my own domain against this?
Yes, free and without an account. The scanner returns both scores out of 100, every component and its points, and the exact DNS record, file or tag to publish for each finding. Enter a domain at astraverify.com.