Do software companies block ChatGPT and Perplexity from their websites?

Almost no software company blocks ChatGPT or Perplexity on purpose. Of 1,586 software company websites I scored, 4 (0.3%) disallowed an AI search bot, Googlebot or Bingbot in robots.txt, and 6 (0.4%) returned a server error on robots.txt. About one in eight still had a hard access problem: 201 sites (12.7%), mostly homepages that did not load for an automated request and pages with no text without JavaScript. 160 (10.1%) showed possible bot blocking at the firewall or CDN, which my test cannot confirm. On 22 October 2026, version 2 of my AI search readiness check went through 1,601 software company websites and scored 1,586.

By Ronan Liedmeier, fractional head of growth. Last updated 23 October 2026.

A search bot is the crawler an AI engine uses to find pages and link them in its answers, such as OpenAI's OAI-SearchBot or PerplexityBot. A training bot collects content that may be used to train models, such as OpenAI's GPTBot or Anthropic's ClaudeBot.

This article covers the Access checks from my 1,586-site study of software company websites.

How many software company websites block AI search bots in robots.txt?

Four. Of the 1,586 software company websites scored, 4 (0.3%) had a robots.txt that disallows OAI-SearchBot, PerplexityBot, Bingbot or Googlebot for the whole site.

Another 6 (0.4%) returned a server error on robots.txt, which leaves a crawler unable to read the rules (scored partial). The remaining 1,576 (99.4%) had no robots.txt problem for those four bots. Across the sample the robots.txt check cost 32 points out of 51,121 lost on all checks, the smallest total of any check.

The check does not score training bots (GPTBot, ClaudeBot, Google-Extended) in robots.txt, so I have no count of software companies that opt out of training. Opting out is a business decision, and OpenAI's docs treat it as separate from search.

What is the difference between search bots and training bots?

Search bots fetch pages so an engine can show and link them in answers. Training bots collect content that may go into model training, and each company documents them as separate user agents with their own robots.txt rules.

OpenAI is the most explicit: "Each setting is independent of the others" (OpenAI). Perplexity says PerplexityBot "is not used to crawl content for AI foundation models" (Perplexity). Anthropic ties search visibility to Claude-SearchBot and Claude-User, and training to ClaudeBot (Anthropic).

To opt out of training, disallow GPTBot and ClaudeBot and leave OAI-SearchBot, PerplexityBot and Claude-SearchBot allowed. OpenAI says robots.txt changes take about 24 hours to reach its search systems, Perplexity up to 24 hours. Their user-triggered fetchers, ChatGPT-User and Perplexity-User, act when a person asks a question, and both companies say robots.txt may not apply to them.

If almost nobody blocks the bots, why does one in eight sites have an access problem?

Most access problems come from how a site is built and protected. 201 of the 1,586 software company websites (12.7%) had at least one hard problem: a homepage that did not load for the checker, no readable text without JavaScript, a robots.txt block, or a noindex or nosnippet tag on the homepage.

The five Access checks, base 1,586 scored sites:

Counting partials and possible firewall blocking too, 391 sites (24.7%) had some access issue.

Access is still the strongest of the four groups: a mean 93.1% of applicable points (median 100%), and 5.0% of all points lost (2,574 of 51,121).

Why do 105 software company homepages not load for an automated request?

105 of the 1,586 homepages (6.6%) did not return HTTP 200 over HTTPS to a plain request from my checker. Many of those sites block automated requests in general, through bot protection that refuses anything that does not look like a browser.

Those 105 sites averaged 21.7 out of 100 (median 13), against 66.9 (median 69) for the 1,481 whose homepage loaded. Most of that gap is the block itself: the checker sees a block or loading page, so the content checks fail too.

Whether OAI-SearchBot or PerplexityBot get the same treatment from their own IP addresses is something this result cannot show, so treat a failed load as a reason to check your logs.

Is a firewall or CDN blocking AI crawlers without anyone deciding to?

Possibly, on about one in ten software company websites. My bot-blocking test flagged 160 sites (10.1% of all scored): 108 refused or thinned out requests sent with AI bot user agents while serving a browser-like request, and 52 refused the browser-like request as well.

On the 1,481 sites whose homepage loaded, 105 (7.1%) showed possible bot-only blocking and 5 (0.3%) refused the browser request too. That 105 is a different group from the 105 homepages that did not load.

The test sends OAI-SearchBot, PerplexityBot, GPTBot and ClaudeBot user agents and flags a site when the bot request is refused (HTTP 401, 403, 429 or 503), gets a challenge page, or gets under 30% of the text the browser-like request gets.

Why the test can raise a false alarm

The test sends those user agents from a residential IP address rather than OpenAI's or Perplexity's networks. Both companies publish their crawler IP ranges (openai.com/searchbot.json and www.perplexity.com/perplexitybot.json), and Perplexity recommends firewall rules that "combine both User-Agent string matching and IP address verification" (Perplexity). A firewall set up that way sees my request as a fake OAI-SearchBot and refuses it, correctly, while the real crawler gets through.

Cloudflare works on the same principle. Its docs describe a verified bot as one confirmed through a published IP list or a cryptographic Web Bot Auth signature, and say that historically verified bots "have been excluded in default bot configurations across all plans" (Cloudflare). A Cloudflare site that refused my fake PerplexityBot may well serve the real one.

The same page says customers can now set their own AI bot policies, so the opposite case is real too: a blocking rule someone switched on in a CDN dashboard that marketing may not know exists. My test cannot tell the two apart, hence "possible". Steps 3 and 4 at the end show how to tell for your own site.

How much of the access problem is JavaScript?

JavaScript is the costliest Access check, with 1,242 points lost. 170 of all 1,586 software company websites (10.7%) failed the no-JavaScript text check and 74 (4.7%) were partial; on the 1,481 whose homepage loaded, 87 (5.9%) failed and 53 (3.6%) were partial.

It matters because, in Vercel's analysis of crawler traffic on its network, "none of the major AI crawlers currently render JavaScript" (Vercel). A page that builds its text in the browser looks empty to them. Sites that failed averaged 25.0 out of 100 against 69.4 for sites that passed, partly because a page with no readable text also fails most content checks.

The breakdown and the fixes are in software company websites without JavaScript.

Do noindex and nosnippet tags keep software companies out of AI answers?

For Google's AI features they can, and 39 of the 1,586 software company homepages (2.5%) carry one. Another 44 sites (2.8%) had the tag only on some sampled pages, and 1,503 (94.8%) were clean.

Google's robots meta tag documentation says nosnippet "will also prevent the content from being used as a direct input for AI Overviews and AI Mode", and that max-snippet:0 is equivalent to nosnippet (Google Search Central). A noindex keeps the page out of search results altogether. If nobody can say why the tag is there, remove it from the template.

What does each access problem do, and how do you fix it?

Each problem has its own fix, listed below, and its own owner: robots.txt and meta tags sit with whoever runs the site, firewall rules with whoever runs hosting or the CDN.

Does fixing access get a software company cited?

No access fix gets a site cited by itself. In my 209-vendor test, the checks that predicted which vendor an answer cited were FAQ sections, named authors with an about page, article depth, comparison tables and visible dates. HTTPS and robots.txt could not be tested because every vendor passed, the JavaScript check was weak and not significant, and the bot-versus-browser check showed nothing.

Access is the floor: an engine cannot quote a page it cannot fetch. After that, see the formats engines quote and the guide to getting cited.

How I ran the check

The sample is software publishers (NAICS 51321 or 5112) with 11 to 200 employees, founded 2015 or later, headquartered in 13 countries: the US plus 12 in Europe (UK and Ireland, DACH, Benelux, Nordics). I pulled them from Apollo's company search on 22 October 2026, with agencies, consultancies, PR firms, online shops and staffing firms excluded by NAICS code. 65 were dropped while pulling (no domain, mislabelled or out of scope) and 32 removed before the run: 12 I had already contacted and 20 very large brands. Country comes from the query's HQ filter, unverified per company.

Of 1,601 attempted, 1,586 were scored, including 3 partial scans that ran out of the 45-second budget. 15 failed, 12 because the domain did not resolve. The run took place on 22 October 2026, 18:03 to 18:28 Pacific time, with the same scoring code as the free readiness check, as a local batch on a residential connection. Each site was checked once, with a median of 9 URLs read per site.

Limits: bot user agents were sent from a residential IP, so A4 is possible blocking only, and a homepage that refused my checker may still serve real crawlers. One check per site means a brief outage counts as a failure. Training bots were not scored in robots.txt. A readiness score is not a citation count, and this one covers software companies with 11 to 200 staff only.

What I'd do

Run these on your own software company site, top to bottom:

1. Request https://yourdomain.com/ with curl and confirm HTTP 200. A 403 or a challenge means your bot protection refuses automated requests; look there first.
2. Open yourdomain.com/robots.txt. Confirm it returns 200 and that nothing disallows "/" for OAI-SearchBot, PerplexityBot, Googlebot, Bingbot or User-agent: *. To opt out of training, add separate rules for GPTBot and ClaudeBot.
3. Compare a browser user agent with a bot user agent, then repeat with PerplexityBot. A different status code means you should investigate:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourdomain.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yourdomain.com/

4. Filter your server or CDN firewall logs for OAI-SearchBot and PerplexityBot and compare source IPs with openai.com/searchbot.json and www.perplexity.com/perplexitybot.json. Requests from those ranges getting a 200 mean step 3 was a false alarm. A 403, or no requests from those ranges at all, means the block is probably real: add an allow rule matching user agent and IP together.
5. View the raw source of your homepage and two content pages and check the main text is there. If it is missing, read the JavaScript article before anyone proposes a rebuild.
6. Search the homepage source and response headers for noindex, nosnippet and max-snippet. Remove any you cannot explain.
7. Run the free readiness check for all five Access checks plus the rest of the 22, and keep the result as next quarter's baseline.

If these steps turn up a block nobody at your software company owns, that is the kind of problem I sort out as a fractional head of growth: find who controls the rule, fix it, then build the pages engines quote. Start with the free readiness check. The AI Visibility Snapshot is the paid next step: ten prompts in your category, your site scored on the 22 checks, and a one-page readout in three working days.

Re-check by 23 January 2027: re-run the Access checks on the same 1,586 sites, add a run from a cloud network beside the residential one to separate firewall false alarms from real blocks, and re-read OpenAI's, Perplexity's and Anthropic's crawler docs for new user agents or changed rules.