How to check whether an AI assistant can read your website

Opening your site in a browser tells you nothing about this. You have JavaScript, cookies, a real browser fingerprint and probably a session — an assistant has none of those. It makes one plain HTTP request, reads whatever comes back, and gives up. So the only honest test is to ask for your page the way it does.

Everything below runs in a terminal with curl, which is already installed on macOS and Linux and ships with Windows 10 and later. Replace your-site.com throughout.

Does it answer at all?

The most complete failure is the one nothing tells you about: the page is up for people and refused for everything else. In our sample of 293 well-known sites, 45 of 293 (15%) refused a plain request outright.

Copycurl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://your-site.com | head -1

You want HTTP/2 200. A 403 or 503 means a protection layer answered instead of your site, and what to do about that is its own subject. A 301 or 302 is fine as long as it lands somewhere; add -L to follow it.

Is there any text in the response?

This is the check that surprises people. If your site renders in the browser, the HTML that arrives may contain no words at all — an empty container and a script tag. Count the paragraphs.

Copy# Roughly how many words arrive before any JavaScript runs
curl -s https://your-site.com | sed -e 's/<[^>]*>/ /g' | wc -w

The median site in our sample returns about 673 words. Under a hundred means an assistant has essentially nothing to read, and no amount of other work matters until that is fixed. Only 11 of 293 (4%) of the sample were this bad, but they were catastrophically bad rather than mildly so.

Does the page say what it is?

A title and a description are how a page introduces itself in every context where it is quoted rather than opened. They are also the two easiest things on this list to get right.

Copycurl -s https://your-site.com | grep -iEo '<title>[^<]*|<meta name="description" content="[^"]*'

A title under about sixty characters, and a description between 120 and 160. Longer gets truncated where it is read; shorter is not a summary of anything.

Is there structured data?

Structured data is the only part of a page an assistant consumes without interpreting it. Everything else it has to infer, and inference is where it invents things about you.

Copycurl -s https://your-site.com | grep -c 'application/ld+json'

Zero is the common answer and the expensive one. One block declaring who you are is the floor; the page-appropriate type on top of it — Product and Offer on pricing, FAQPage on an FAQ, Article on a post — is what lets an assistant answer "how much is it" without reading your prose at all.

Which crawlers does your robots.txt turn away?

Read your own file and check it against what each crawler is for. The distinction that matters is that training crawlers cost you nothing to block and answer engines cost you the answer.

Copycurl -s https://your-site.com/robots.txt

Every token, what it does and what blocking it costs is on the AI crawler reference. The specific trap: blocking GPTBot does not remove you from ChatGPT answers, and blocking OAI-SearchBot does.

What does the whole thing actually cost to read?

An assistant pays for your page in tokens and has a budget. If most of what arrives is markup, scripts and serialised state, the content competes with the furniture for room in the context window. Across our sample the median page cost about 2719 tokens, of which 30% was navigation and boilerplate rather than the thing the page is about.

Copy# Bytes down the wire, versus bytes of readable text
curl -s https://your-site.com | wc -c
curl -s https://your-site.com | sed -e 's/<[^>]*>/ /g' | tr -s ' ' | wc -c

A readable share under about 5% means the page is mostly scaffolding. That is worth knowing but it is the last thing to fix, not the first.

What this does not tell you

These six checks find the failures that stop a page being read. They do not tell you whether what was read is enough to answer a question about you — whether an assistant could say what you sell, what it costs, where you are and what happens if somebody wants their money back. That is a harder question and it is the one worth asking next, because a page can pass every check above and still leave an assistant unable to say anything useful about your business.