# Can ChatGPT see my website? A five-minute check anyone can run

**You can find out in about five minutes, and the answer is usually not what the owner expects.** An AI assistant can only mention your business if its crawler can reach your pages, read them as real text, and, for ChatGPT search specifically, find them in Bing's index. Each of those is checkable with the crawler names the companies publish themselves, and none of it needs a tool you pay for.

## Which crawlers are we actually talking about?

Every major AI company runs more than one crawler, and they do different jobs. One collects training
data. One surfaces your site inside that company's own search or answer product. One fires only when a
live user asks a question that needs your page. Blocking the first does nothing to the second, and it is
the second that decides whether you appear in an answer. OpenAI's own crawler documentation puts it
plainly: [sites that are opted out of OAI-SearchBot](https://developers.openai.com/api/docs/bots) "will not be shown in
ChatGPT search answers."

| Company | Crawler | What it is for |
| --- | --- | --- |
| OpenAI | OAI-SearchBot | ChatGPT search answers. This is the one that matters for being cited. |
| OpenAI | ChatGPT-User | A live user's question. OpenAI says robots rules "may not apply" because a person asked. |
| OpenAI | GPTBot | Training data. Blocking it does not remove you from ChatGPT search. |
| Anthropic | Claude-SearchBot, Claude-User, ClaudeBot | Search quality, live user fetch, training. All three honour robots.txt. |
| Perplexity | PerplexityBot, Perplexity-User | Perplexity answers, and a live user fetch that generally ignores robots.txt. |
| Google | Googlebot, Google-Extended | Search and AI Overviews, and a separate token for Gemini training that does not affect Search. |
| Microsoft | bingbot | Bing, Copilot, and the index ChatGPT search draws on. |

Sources: [OpenAI, Bots and crawlers](https://developers.openai.com/api/docs/bots); [Anthropic, Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518); [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers); [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers).

## Step one: read your own robots.txt

Open yourdomain.ca/robots.txt in a browser. You are looking for any of the
names above followed by Disallow: /. Two things surprise people here. The
first is that a web designer or a plugin often added blocks years ago for reasons nobody remembers. The
second is that on Cloudflare, a setting called
[managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) can insert a block for AI crawlers *above*
the file you wrote, so the live file is not the file in your project. Read the live one.

## Step two: fetch the page the way the crawler does

Permission is not access. A firewall, a bot-protection rule or a challenge page can refuse a crawler
that your robots file invited, and no ranking report will ever show it. Google's guidance on AI
features lists, as the first thing to get right,
[ensuring that crawling is allowed](https://developers.google.com/search/docs/appearance/ai-features) "in robots.txt, and by any CDN or hosting
infrastructure." OpenAI goes further and publishes the IP ranges its crawlers use so a site owner can
allow them explicitly.

The check is one command per crawler, run from any computer with a terminal:

```
curl -sS -o /dev/null -w "%{http_code} %{size_download}\n" -A "OAI-SearchBot/1.4" https://yourdomain.ca/
```

Run it once with no -A to get the browser's number, then once per crawler
name. You want the same status and the same byte count every time. A 403, a
503, or a body a fraction of the browser's size means that assistant is being
turned away at the door. On our own domain, twelve crawlers currently receive exactly what a browser
receives, and the [readout on the AI Visibility Pass page](https://bluecraneworks.ca/solutions/ai-visibility.html) shows every row.

## Step three: is the text actually in the page?

Right-click the page and choose View Source. If you can read your own sentences in there, good. If
you see a nearly empty shell and a pile of script tags, the words are painted in by JavaScript after the
page loads, and most crawlers never see them. Bing names this outright in its
[Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a): "hiding critical content behind client-side
rendering" is on the list of things to avoid, and "content that cannot be reliably rendered may not be
indexed or selected for grounding results." Google is gentler but says the same thing: server-side or
pre-rendering is "still a great idea" because "not all bots can run JavaScript."

## Step four: is the site in Bing at all?

Search Bing, not Google, for site:yourdomain.ca. ChatGPT's web search
runs on Bing's index with OpenAI's own crawler on top, so a site Bing has never indexed cannot be cited
there however well it does on Google. If Bing shows nothing, the fix is registration and a sitemap, and
it is the subject of [why your website needs Bing to appear in ChatGPT](https://bluecraneworks.ca/articles/why-your-website-needs-bing-to-show-up-in-chatgpt).

## Step five: look for the two silent blocks

In that same page source, search for noarchive and
nosnippet. Either one, in a meta tag or in an HTTP header, removes the page
from Copilot citations and from Google's AI answers while leaving every ranking tool green. They are
common, they are usually accidental, and they are explained in
[the two settings that hide a website from AI answers](https://bluecraneworks.ca/articles/noarchive-nosnippet-remove-your-site-from-ai-answers).

## What does a pass look like?

Every crawler receives the full page. The words are in the HTML. Bing has the site. Nothing on any
page says noarchive or nosnippet. That is the whole floor, and most small-business websites fail at
least one step without anyone knowing. Google states there are
[no additional requirements](https://developers.google.com/search/docs/appearance/ai-features) to appear in AI Overviews or AI Mode beyond being
indexed and eligible for a snippet, and that "you don't need to create new machine readable files, AI
text files, or markup" to get there. Bing's guidelines add the honest limit that applies to everyone:
"SEO does not guarantee rankings or traffic, and GEO does not guarantee grounding or citations in AI
experiences." Nobody can buy the citation. What you can do is stop being the business the machine
cannot read.

## Want this checked and fixed on your site?

The AI Visibility Pass runs every check in this article against your live site, fixes what it
finds, and hands you the proof. Fixed quote, written guarantee. On a managed website it is included
from launch.

[See the AI Visibility Pass](https://bluecraneworks.ca/solutions/ai-visibility.html)
[Managed websites from $40 a month](https://bluecraneworks.ca/solutions/websites.html)

## Next, from Blue Crane

- [**The two settings that hide a site from AI answers**Step five of the check, in full](https://bluecraneworks.ca/articles/noarchive-nosnippet-remove-your-site-from-ai-answers)

- [**Why your website needs Bing to appear in ChatGPT**Step four of the check, in full](https://bluecraneworks.ca/articles/why-your-website-needs-bing-to-show-up-in-chatgpt)

- [**The AI Visibility Pass**Every check in the article, run and fixed for you, under a written guarantee](https://bluecraneworks.ca/solutions/ai-visibility.html)

---

Source: https://bluecraneworks.ca/articles/can-chatgpt-see-my-website