robots.txt GPTBot: check if AI crawlers can read your site
To check robots.txt GPTBot access, open yourdomain.com/robots.txt in a browser and look for a line reading User-agent: GPTBot followed by Disallow: /. If it is there, OpenAI's training crawler is shut out of your whole site. If it is missing, you have cleared one gate out of several. Your firewall and the way your pages render can also keep ChatGPT, Gemini and Perplexity away from a site whose robots.txt looks fine, and your server logs are where you confirm any of it.
Which AI crawlers to look for
OpenAI alone sends three bots, and each does a different job. Blocking one leaves the others untouched, so read the file for every name below.
| Crawler | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects pages for model training |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a person asks ChatGPT to open it |
| PerplexityBot | Perplexity | Builds Perplexity's search index |
| Google-Extended | Opt-out token for Gemini use; Googlebot still governs Search and AI Overviews |
Also look for a blanket User-agent: * rule with Disallow: /. Staging sites often carry one, and it sometimes survives the launch. If a plugin or CDN setting was meant to block AI crawlers, find out who switched it on and what exactly it covers.
What a robots.txt GPTBot rule does and does not control
A robots.txt GPTBot entry only governs OpenAI's training crawler. Whether you can appear in ChatGPT's search answers depends on OAI-SearchBot, and whether ChatGPT can open your page for a user depends on ChatGPT-User. Plenty of owners disallow GPTBot to keep their content out of training and allow the other two. That is a legitimate choice, as long as you make it on purpose.
Google works differently. AI Overviews draw on pages Googlebot can index. Ahrefs found that 38% of AI Overview citations come from the top 10 results and 31% from positions 11 to 100 (Ahrefs, March 2026). If Googlebot cannot fetch a page, it has no chance at either group. Blocking Google-Extended leaves your place in Search alone and tells Google you opt out of Gemini use.
Check your firewall and CDN
Robots.txt is a request. Your firewall is enforcement. Bot protection at Cloudflare, Akamai, Sucuri or your host can return a 403 to unfamiliar user agents regardless of what robots.txt says.
You can test this from a terminal. Run curl -I -A OAI-SearchBot https://yourdomain.com/ and then the same command with PerplexityBot. Compare the result with a request using a normal browser user agent. A 200 is what you want. A 403, a 429 or a challenge page means the door is shut. If you get a 403, ask whoever manages your CDN or host which rule fired and whether it can be narrowed so the crawlers you want are let through.
Test a product page, a guide page and your home page, because rules are often set per path. One caveat: some firewalls verify real bots by IP range, so a curl test from your laptop can differ from what the actual crawler gets. Treat it as a quick screen and let the logs settle it.
Check that the content is in the HTML
Assume AI crawlers do not run your JavaScript. Fetch the raw page with curl and search the output for a product name, a price or the first sentence of a guide. If the text is missing, a crawler sees an empty shell.
This bites online stores hardest. Prices, variant names, stock status and reviews are often injected by script. If they are absent from the raw HTML, fix that before anything else on this list. Also look for a stray noindex tag, a canonical pointing somewhere unexpected, and pages behind a login or cookie wall. Classic SEO hygiene still sets the floor here.
Confirm with server logs
Logs show what happened, where the other tests only show what should happen. Search your access logs for the user agent strings GPTBot, OAI-SearchBot, ChatGPT-User and PerplexityBot. On a Linux server that is a grep -i gptbot on the access log. If you have no raw logs, your CDN dashboard usually has a bot traffic report. Note the date of the last hit from each bot, so you can tell a one-off visit from a steady pattern.
Read the status codes next to each hit. Repeated 200s on money pages and guides mean the crawler can read you. A run of 403s or 429s points to the firewall. No hits over a month proves less on its own, because a quiet site can simply go unvisited, but zero hits plus a failed curl test is a clear signal.
Skip llms.txt and check your listings
Adding an llms.txt file is not a fix for any of this, and Google discourages the file.
If you run several locations, crawler access is only part of the picture. Gemini grounds local answers on Google Maps, while ChatGPT and Perplexity use Bing Places and Yelp (per the platforms' own sourcing). A readable site with a stale Bing Places listing can still lose the recommendation.
Once the crawlers can get in, check whether the engines cite you. The free LynkAI scan shows where you stand in ChatGPT, Gemini and Perplexity.