Skip to main content

When your site blocks AI crawlers

A firewall, CDN, robots.txt rule or challenge can prevent a particular reader from opening a page. Fixing access removes that obstacle; it does not guarantee a citation, recommendation or sale. A Clione report marked Blocked describes the request our crawler could not complete, rather than proving that every AI reader is blocked.

What the blocked report tells you​

Keep the job ID, blocked URL, HTTP status and block type from the report notice. A 401, 403, 429 or 503, a recognised challenge page, or a robots.txt refusal stops this report. A challenge can arrive with HTTP 200; a robots.txt refusal can happen before the page is requested and therefore has no page HTTP status. A 503 may also reflect temporary unavailability: use the response and logs to establish the cause.

Score my store sends ClionePulse/0.1 (AI Shelf Report) and checks robots.txt for ClionePulse. It is separate from the vendors’ readers below. Ask Clione support for current report-crawler egress information before creating an IP-scoped exception; do not substitute an AI vendor’s IP list or your own test IP.

A report marked Failed indicates a processing error. Our team is alerted; changing your firewall is not the appropriate response to that status.

Search, user requests and training are different​

ReaderPurposeCurrent operator guidance
OAI-SearchBotChatGPT search discoveryOpenAI crawlers
ChatGPT-UserSome user-requested page readsSame guidance; robots.txt may not apply to these requests
GPTBotModel-training collectionSeparate from the search crawler
Claude-SearchBot / Claude-User / ClaudeBotSearch / user requests / trainingAnthropic crawler controls
PerplexityBot / Perplexity-UserSearch / user requestsPerplexity crawlers and current IP lists
GooglebotGoogle Search crawlingGoogle crawler reference
BingbotBing crawlingMicrosoft bot verification

Choose which uses you permit. Separate controls are not interchangeable: Google-Extended is a policy token, not a distinct HTTP user agent, and affects both training and some Gemini grounding uses. It does not determine inclusion or ranking in Google Search. See Google’s reference.

Five checks to find the cause​

  1. Read your robots.txt. Check wildcard and reader-specific groups, including rules added by a CDN. Keep existing private-path restrictions. A more-specific group can change which rules apply. Do not replace the whole file with a generic template.
  2. Make a GET request. Set the full page URL and user agent yourself, then inspect both the headers and HTML:
: "${PAGE_URL:?Set the full page URL to test}"
: "${CRAWLER_USER_AGENT:?Set the reader user agent to test}"
curl --silent --show-error --location --max-time 30 \
--user-agent "$CRAWLER_USER_AGENT" \
--dump-header crawl-headers.txt --output crawl-page.html "$PAGE_URL"

Repeat for home, category and product pages. 200 alone is insufficient: inspect the body for a challenge, login screen or empty page. A request with a copied user agent comes from your IP, not the vendor’s. It is a diagnostic hint, not identity verification or proof of how that vendor is treated.

  1. Try an assistant’s page-reading feature. A successful read demonstrates that particular request. A refusal can have other causes, including tool availability or the product’s policy; correlate it with server logs before blaming your firewall.
  2. Inspect server/CDN logs. Match time, URL, user agent, source IP, response and rule/action. No recorded hits means not observed, not proof of blocking. Actual verified blocked requests establish which traffic your rule refused.
  3. Inspect content without JavaScript. Check the downloaded HTML for product name, public price, availability, description and useful specifications. A successful HTTP request can still return no usable product facts. See product-data delivery.

Common causes​

CauseWhere to look
AI-crawler policy or managed robots rulesCDN security settings and the actual served robots.txt
Bot challenge or CAPTCHACloudflare, Akamai, Imperva, DataDome or another bot manager
Rate limitWAF, hosting or a security plugin; look for 429 and rule evidence
Country or network restrictionFirewall rules targeting geography, ASN or data-centre traffic
Login, password page or age gateStorefront access settings
JavaScript-only product contentTheme or front-end rendering

Fix the rule that actually blocked the request​

Change one relevant rule at a time, then repeat the same checks. Keep authentication and protection for private store areas.

Cloudflare​

Use Security Events to identify the responsible product and rule. Review AI Crawl Control, managed robots.txt, verified-bot handling and rate limits for the readers you intend to permit. Scope exceptions using verified identity or current official IP information rather than trusting a user-agent string alone.

Bot Fight Mode and Super Bot Fight Mode differ: a custom WAF Skip rule cannot bypass Bot Fight Mode. Super Bot Fight Mode supports scoped exceptions. Ask your administrator to choose the appropriate control; do not assume a Skip rule fixed the request or disable unrelated protection. Cloudflare’s false-positive guidance explains the limitation.

Other firewalls and bot managers​

For Akamai, Imperva, DataDome, Fastly, Sucuri, AWS WAF or Vercel Firewall, give your administrator the blocked request and rule evidence. Use the operator’s supported verification process and make a narrow exception for the intended public content. Recheck rate limits and geography/network rules as well as bot classification.

Storefront checks​

  • Shopify: inspect bot/security apps, custom robots.txt.liquid rules, storefront passwords and age gates. Do not assume the platform or a particular app is the cause without logs.
  • BigCommerce: inspect the served robots.txt, security apps and any CDN in front of the custom storefront domain.
  • WooCommerce / WordPress: inspect security-plugin bot lists and rate limits, the host’s firewall and coming-soon/private-site settings.
  • PrestaShop: inspect the generated robots.txt, security modules, maintenance/access settings and the hosting firewall.
  • commercetools / headless: the agency or front-end owner controls the storefront renderer and edge rules. Check both.

robots.txt communicates crawler preferences; it is not access control and cannot undo a WAF refusal. Preserve your existing policy and use the operator references above when changing individual groups. Do not treat a training choice as a universal switch for search or live page reading.

Verify the reader, then keep the rules current​

Use official operator documentation for current ranges and verification procedures. Combine identity checks with the intended public paths. User agents can be copied, and IP ranges can change. For DNS verification, check the operator’s required hostname suffix and forward resolution rather than trusting a reverse lookup alone.

Perplexity publishes separate bot/user ranges. OpenAI publishes ranges for its individual agents. Anthropic documents its crawler policy and identity information. Google and Microsoft document crawler verification. Do not paste fixed IP addresses from an old article into a permanent rule.

Once access works, check the content​

Serve useful product text, specifications, FAQs and image descriptions in the HTML the server returns. Keep your sitemap accurate and responses stable. JSON-LD remains useful for SEO; it is not a promise that a live AI reader receives or uses that markup. Access, delivery, citation and sales are different measurements.

Confirm the change​

  1. Repeat the same GET checks for home, category and product pages and inspect their content.
  2. Confirm intended, verified reader requests and responses in your own logs.
  3. Check real page-reading results separately; do not infer them from a copied user agent.
  4. Request a fresh report through the form you used originally. If it is still blocked, share the job ID, blocked URL, HTTP status and block type with support. Do not share credentials or session cookies.

Frequently asked questions​

Will allowing AI crawlers hurt my Google rankings?

Access for another crawler is separate from Googlebot access. Keep your Googlebot rules and normal SEO checks. Allowing access does not guarantee rankings, citations, recommendations or sales.

Can I allow search without allowing model training?

Some operators publish separate search, user-request and training controls. Check each operator’s current documentation. Google-Extended also controls some Gemini grounding uses, so it is not a training-only switch.

Will crawlers slow my site down?

Measure request volume and response times in your own logs. If load is a problem, agree suitable rate limits with your hosting provider instead of assuming that every crawler should be allowed without limits.

Does robots.txt alone fix a blocked request?

Only when robots.txt was the cause. It cannot override a firewall, CAPTCHA, login wall or rate limit. Some user-request fetchers do not apply robots.txt in the same way as automatic crawlers.

Do I need an llms.txt file?

No. It is not a Clione access requirement, setup recommendation or readiness-score signal. Fix request access and useful content in the served HTML first; existing experimental endpoints do not prove that an AI reader uses them.

My competitor blocks AI crawlers. Should I?

Their policy does not establish what works for your store. Choose your own access and training policy, verify the requests you permit, and measure actual results rather than assuming a competitor’s citations or sales.