Count AI crawlers with a Cloudflare Worker

Plans: every plan. Platforms: BigCommerce and Shopify stores whose domain is on your own Cloudflare account. On WordPress / WooCommerce you do not need this: the Clione plugin counts the same thing (Settings → Clione FAQ → Count AI crawler visits).

AI assistants can only use what their crawlers have read. Clione shows, for every product, category, collection and page, which AI crawlers requested it and when — next to its enrichment versions — so you can see whether a change was picked up. That needs a count taken on the server: these crawlers do not run JavaScript, so an analytics tag never sees them.

On BigCommerce and Shopify the page is rendered on the platform's own servers, which run no Clione code. The one server-side place left is in front of the platform: a Cloudflare Worker on your domain. That is what this page installs.

What it counts — and what it does not

Per page (URL path, without the query string) and per day:

Crawler	Who	What it is
GPTBot	OpenAI	collects pages for model training
OAI-SearchBot	OpenAI	indexes pages for ChatGPT search
ChatGPT-User	OpenAI	fetches a page live because someone asked ChatGPT
PerplexityBot	Perplexity	indexes pages for Perplexity answers
Perplexity-User	Perplexity	fetches a page live for a Perplexity user
ClaudeBot	Anthropic	collects pages for model training
Claude-User	Anthropic	fetches a page live because someone asked Claude
Bingbot	Microsoft	indexes pages for Bing and Copilot

…plus the total number of page requests, so you can read each number as a share. Only HTML pages that answered 200 count; images, scripts and errors do not.

• No IP address is read, stored or sent. Only the user agent is looked at, and it is reduced to one of the eight names above or to nothing.
• Your pages are never slowed down or changed. The Worker passes every request to your store and returns the store's answer untouched; counting happens after the answer has gone out, and if anything in the Worker fails, Cloudflare falls back to your store as if the Worker were not there.
• Clione being down never affects your store. The counts are kept in your Cloudflare account and sent once a day; a day that could not be sent is sent on the next run (up to 14 days back).
• The counts are a floor: a few requests counted in the seconds before Cloudflare recycles a Worker instance can be lost. Nothing is ever counted twice.
• Every page is counted as it comes, including account, checkout and cart pages. So in rare cases a stored page address can contain an order number (for example when a crawler fetched an order page in a customer account). The host and the query string are never kept.
• Clione keeps the counts for 13 months, then deletes them automatically, day by day.

Before you start: is your domain really yours on Cloudflare?

The Worker runs only on a Cloudflare zone you control. In Clione, open your store → Overview → AI-crawler visits → Check whether my domain can run the Worker. Clione reads your domain's nameservers and tells you one of:

Result	What it means
On Cloudflare	Your domain's nameservers are Cloudflare's. Go ahead.
Shopify / BigCommerce manages the DNS	The platform runs your domain's DNS, so there is no Cloudflare zone of yours to put the Worker on. The Worker cannot run here.
A myshopify.com / mybigcommerce.com address	That address lives in the platform's own zone. The Worker cannot run there — it needs your own domain.
Another DNS provider	Your domain is not on Cloudflare. Move its DNS to your Cloudflare account first, or the Worker cannot run.

Two more things the check cannot see for you:

1. The store's DNS record must be proxied (the orange cloud in Cloudflare → DNS). A "DNS only" (grey cloud) record sends visitors straight to the platform and the Worker never sees them. Proxying a Shopify or BigCommerce storefront through your own Cloudflare zone is a supported Cloudflare setup ("orange-to-orange"); follow your platform's guidance before you switch it.
2. The Cloudflare account must be yours, because you are the one adding the Worker to the route.

Install it

You need: your Cloudflare account, Node.js on your computer, and ten minutes.

1. Create the key in Clione. Settings → API Keys → Generate key: choose this store, tick only the crawler:hits permission, and copy the key (sk_live_…). It can send this store's crawler counts and nothing else, and revoking it in Clione stops the Worker at once.
2. Get the Worker files from Clione (ask us for the crawler-hits-worker folder): src/ and wrangler.example.toml.
3. Copy wrangler.example.toml to wrangler.toml and fill in:
   • routes — your store's domain followed by /*, and your zone name;
   • CLIONE_API_BASE — the Clione address shown in your dashboard;
   • CLIONE_TENANT_ID — your organisation id, shown in your dashboard;
   • CLIONE_SITE_URL — your store's address, https:// included. It must be the same domain your store has in Clione, or Clione refuses the counts.
4. Log in and add the key as a secret (it never goes in the file):

   npx wrangler login
   npx wrangler secret put CLIONE_API_KEY
5. Deploy:

   npx wrangler deploy

That is all. The Worker counts from the first request; the first day arrives in Clione the next day.

The first day: approve the feed

Counts from a new source wait in quarantine until someone on your team says they are yours. When the first day arrives, your store's AI-crawler visits card says waiting for your approval: an owner or admin clicks Approve this store's crawler counts once, and the following days come in on their own. You can revoke it at any time; Clione then holds new days for approval again.

Check it is working

• In Cloudflare → Workers → clione-crawler-hits → Logs, requests pass through with your store's status codes.
• curl -I https://<your-store-domain>/ returns your store's usual headers — the Worker adds none.
• The day after installing, the store's AI-crawler visits card shows the approval prompt or the first numbers.

Remove it

npx wrangler delete (or delete the Worker in the Cloudflare dashboard), then revoke the key in Clione → Settings → API Keys. Your store is unaffected either way.

Troubleshooting

Symptom	Cause
Nothing arrives after two days	The DNS record is not proxied (grey cloud), the route does not match your domain, or CLIONE_SITE_URL is a different domain than the store's in Clione.
The card still says not measurable	The key is not bound to this store, or it lacks the crawler:hits permission.
Only some pages show visits	Only pages a crawler actually requested appear, and only 200 HTML answers count. A page with no row on a day the store reported had zero crawler visits — that is a real zero.