AI Crawler and Bot User Agents API

Q: Which user agents should I block to stop AI training?

The tokens to disallow to stop AI training are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent — /api/crawlers/robots-tokens?purpose=ai-training returns them with the exact User-agent: line for each. CCBot is **our** classification rather than the operator's wording — the other 5 labels come from the operator's own page — because Common Crawl's page describes only "an open repository of web crawl data" and never mentions training — its corpus is simply the canonical public training corpus, and the row's note says exactly that. Amazon's Amazonbot is deliberately general rather than ai-training: Amazon states it "is used to improve our products and services" and names training only as a possible further use ("may be used to train Amazon AI models"), so disallow it too if you want that use excluded.

REAL DATA

Known crawler and AI-bot user agents: the robots.txt token each operator publishes, and what a user-agent string you supply actually matches.

Authoritative reference data or standards computation

Base URL: /api/crawlers
Category: Internet & Devices
Capabilities: 3 routes
Last updated: July 30, 2026

Data sources · 4: crawler-user-agents (monperrus), MIT — snapshot 2026-07-30
OpenAI bots documentation
Google crawlers and fetchers documentation

+1 more

Anthropic crawler documentation

GET /api/crawlers List known crawler and AI-bot user-agent patterns Returns a list

All 1500 bundled patterns, ordered by 'pattern' ascending on UTF-16 code units so paging with 'offset' is stable. Filters compose with AND; a combination that matches nothing returns an empty list plus a warning, never a substitute row.

Live requestRuns against the public API

No key required

GET/api/crawlers?tag=ai-crawler&count=100

Request parameters

tagdefault any

Keep only rows the upstream project gives this tag. A row can carry several. The 2 operator-documentation rows carry no tags and are never returned by this filter.

operatordefault any

Keep only rows curated to this operator — the ones whose own documentation states the token.

documented

true keeps only rows whose operator documents the bot itself (operator is not null); false keeps only the uncurated community rows. Combining false with an operator is a 400, because nothing can satisfy both.

search

Case-insensitive substring match against 'pattern', 'robotsToken' and 'operator'. At least 2 characters — a single character matches almost the whole list and is rejected with a 400.

offsetdefault 0

Skip this many matches before 'count' is applied — the list is far longer than count's ceiling of 100, so ?count=100&offset=100 returns the second hundred.

Advanced response options6 options

countdefault 50

Limit the number of returned records (1–100). Defaults to 50.

fields

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

exclude

Return all fields except these (comma-separated).

formatdefault json

Response format: json envelope, ndjson (one record per line) or csv.

prettydefault false

Pretty-print the JSON response.

unwrapdefault false

Drop the envelope: return the raw array/object without data/meta wrapper.

Response

Example parameters are ready. Send the request to inspect the live response.

Route reference

Code, request parameters, response fields and verified examples 11 params · 13 fields · 5 examples

Code samples Ready-to-copy requests in 4 languages

curl JavaScript Python PHP

curl "https://randomapi.dev/api/crawlers?tag=ai-crawler&count=100"

const res = await fetch("https://randomapi.dev/api/crawlers?tag=ai-crawler&count=100");
const { data, meta } = await res.json();

import requests

data = requests.get("https://randomapi.dev/api/crawlers?tag=ai-crawler&count=100").json()["data"]

$json = json_decode(file_get_contents(
  "https://randomapi.dev/api/crawlers?tag=ai-crawler&count=100"
), true);
$data = $json["data"];

Parameters Route-specific request inputs 5

Parameter	Type	Default & allowed	Description
`tag`	enum	default: any allowed: search-engine \| advertising \| feed-reader \| http-library \| social-preview \| archiver \| seo \| monitoring \| scanner \| ai-crawler \| academic \| browser-automation \| any example: ai-crawler	Keep only rows the upstream project gives this tag. A row can carry several. The 2 operator-documentation rows carry no tags and are never returned by this filter.
`operator`	enum	default: any allowed: openai \| anthropic \| perplexity \| google \| apple \| common-crawl \| meta \| amazon \| any example: anthropic	Keep only rows curated to this operator — the ones whose own documentation states the token.
`documented`	boolean	example: true	true keeps only rows whose operator documents the bot itself (operator is not null); false keeps only the uncurated community rows. Combining false with an operator is a 400, because nothing can satisfy both.
`search`	string	allowed: ≤ 64 example: perplexity	Case-insensitive substring match against 'pattern', 'robotsToken' and 'operator'. At least 2 characters — a single character matches almost the whole list and is rejected with a 400.
`offset`	int	default: 0 allowed: 0 – 5000 example: 100	Skip this many matches before 'count' is applied — the list is far longer than count's ceiling of 100, so ?count=100&offset=100 returns the second hundred.

tag enum

Keep only rows the upstream project gives this tag. A row can carry several. The 2 operator-documentation rows carry no tags and are never returned by this filter.

default: any

example: tag=ai-crawler

operator enum

Keep only rows curated to this operator — the ones whose own documentation states the token.

default: any

example: operator=anthropic

documented boolean

example: documented=true

search string

Case-insensitive substring match against 'pattern', 'robotsToken' and 'operator'. At least 2 characters — a single character matches almost the whole list and is rejected with a 400.

allowed: ≤ 64

example: search=perplexity

offset int

Skip this many matches before 'count' is applied — the list is far longer than count's ceiling of 100, so ?count=100&offset=100 returns the second hundred.

default: 0

allowed: 0 – 5000

example: offset=100

Universal parameters Shared response and formatting options 6

Parameter	Type	Default & allowed	Description
`count`	int	default: 50 allowed: 1 – 100 example: 3	Limit the number of returned records (1–100). Defaults to 50.
`fields`	list	example: pattern,patternSource	Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
`exclude`	list	example: addedOn	Return all fields except these (comma-separated).
`format`	enum	default: json allowed: json \| ndjson \| csv example: csv	Response format: json envelope, ndjson (one record per line) or csv.
`pretty`	boolean	default: false example: true	Pretty-print the JSON response.
`unwrap`	boolean	default: false example: true	Drop the envelope: return the raw array/object without data/meta wrapper.

count int

Limit the number of returned records (1–100). Defaults to 50.

default: 50

allowed: 1 – 100

example: count=3

fields list

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

example: fields=pattern,patternSource

exclude list

Return all fields except these (comma-separated).

example: exclude=addedOn

format enum

Response format: json envelope, ndjson (one record per line) or csv.

default: json

allowed: json | ndjson | csv

example: format=csv

pretty boolean

Pretty-print the JSON response.

default: false

example: pretty=true

unwrap boolean

Drop the envelope: return the raw array/object without data/meta wrapper.

default: false

example: unwrap=true

Response schema Fields returned in each record 13

Field	Type	Description	Example
`pattern`	string	The regular expression this row is matched by, verbatim. Matched case-sensitively with RegExp.test, because the upstream patterns encode their own case alternatives ([cC]laude[bB]ot). 13 patterns are anchored to the start of the string, so leading whitespace changes the answer.	[cC]laude[bB]ot
`patternSource`	string	crawler-user-agents (the pinned community list, 1,498 rows) or operator-documentation (a token the operator documents that the community list does not carry, so the pattern was authored from the operator's own page — exactly 2 rows).	crawler-user-agents
`tags`	string[]	The upstream project's own classification, from its fixed 12-value vocabulary, in canonical order. It is the maintainers' judgement, not the operator's self-description. Empty on the 2 operator-documentation rows — the project has not classified them and we do not classify on its behalf, so those rows appear under no tag filter.	["ai-crawler"]
`operator` nullable	string	Company or project that operates this crawler — set only for rows curated against the operator's own published documentation. null on the 1,476 community rows: we do not guess who runs a bot.	Anthropic
`operatorId` nullable	string	The value to pass back as ?operator=. null whenever operator is.	anthropic
`robotsToken` nullable	string	The exact token to write after 'User-agent:', as the operator publishes it. null on every uncurated row — a token derived from a regular expression would be a guess about capitalisation, and a wrong token in your robots.txt is worse than no token.	ClaudeBot
`purpose` nullable	string	What this crawler is for: search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. Our restatement of the operator's published wording, never invented — except on the 2 rows whose own note says the label is ours rather than the operator's (today CCBot and meta-webindexer), which is why the note ships next to it. null when uncurated.	ai-training
`exampleUserAgent` nullable	string	One real User-Agent string, observed in the wild, that this pattern matches — the longest entry the upstream snapshot carries for this row. Paste it into /api/crawlers/detect. Non-null on 118 of the 1,500 rows by a published rule: the rows a curated operator token points at, every row the upstream list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. For any other pattern, paste a line from your own access log instead. Never the operator's own published template — that is fullUserAgentExample on /robots-tokens.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
`referenceUrl` nullable	string	The reference link the community list carries for this crawler, copied and NOT verified: 160 rows have none, 298 are plain http://, 2 are malformed upstream (a missing or truncated scheme) and are still copied byte for byte because inventing a scheme would be inventing provenance, and for a few hundred it points at a third-party bot directory, a forge or a forum thread rather than the operator's own site. Use operatorDocsUrl when you need a citable page.	http://www.google.com/bot.html
`operatorDocsUrl` nullable	string	The operator's own documentation page for its crawlers — curated rows only. This is the URL you can cite.	https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
`verification` nullable	object	How to check a request really came from this operator: { method, url }, where method is published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published, and url is the operator's own list (or null). We never fetch it — it is a URL for you to use. Present on curated rows only.	{"method":"published-ip-ranges","url":"https://claude.com/crawling/bots.json"}
`note` nullable	string	One curated caveat in our own words, where the operator's documentation says something that changes how you use the token. Curated rows only.	Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity.
`addedOn` nullable	string (YYYY-MM-DD)	Date the pattern entered the community list — NOT the date the crawler began operating. null on 55 rows.	2015-04-14

pattern string

The regular expression this row is matched by, verbatim. Matched case-sensitively with RegExp.test, because the upstream patterns encode their own case alternatives ([cC]laude[bB]ot). 13 patterns are anchored to the start of the string, so leading whitespace changes the answer.

example: [cC]laude[bB]ot

patternSource string

crawler-user-agents (the pinned community list, 1,498 rows) or operator-documentation (a token the operator documents that the community list does not carry, so the pattern was authored from the operator's own page — exactly 2 rows).

example: crawler-user-agents

tags string[]

The upstream project's own classification, from its fixed 12-value vocabulary, in canonical order. It is the maintainers' judgement, not the operator's self-description. Empty on the 2 operator-documentation rows — the project has not classified them and we do not classify on its behalf, so those rows appear under no tag filter.

example: ["ai-crawler"]

operator string nullable

Company or project that operates this crawler — set only for rows curated against the operator's own published documentation. null on the 1,476 community rows: we do not guess who runs a bot.

example: Anthropic

operatorId string nullable

The value to pass back as ?operator=. null whenever operator is.

example: anthropic

robotsToken string nullable

The exact token to write after 'User-agent:', as the operator publishes it. null on every uncurated row — a token derived from a regular expression would be a guess about capitalisation, and a wrong token in your robots.txt is worse than no token.

example: ClaudeBot

purpose string nullable

What this crawler is for: search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. Our restatement of the operator's published wording, never invented — except on the 2 rows whose own note says the label is ours rather than the operator's (today CCBot and meta-webindexer), which is why the note ships next to it. null when uncurated.

example: ai-training

exampleUserAgent string nullable

One real User-Agent string, observed in the wild, that this pattern matches — the longest entry the upstream snapshot carries for this row. Paste it into /api/crawlers/detect. Non-null on 118 of the 1,500 rows by a published rule: the rows a curated operator token points at, every row the upstream list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. For any other pattern, paste a line from your own access log instead. Never the operator's own published template — that is fullUserAgentExample on /robots-tokens.

example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

referenceUrl string nullable

The reference link the community list carries for this crawler, copied and NOT verified: 160 rows have none, 298 are plain http://, 2 are malformed upstream (a missing or truncated scheme) and are still copied byte for byte because inventing a scheme would be inventing provenance, and for a few hundred it points at a third-party bot directory, a forge or a forum thread rather than the operator's own site. Use operatorDocsUrl when you need a citable page.

example: http://www.google.com/bot.html

operatorDocsUrl string nullable

The operator's own documentation page for its crawlers — curated rows only. This is the URL you can cite.

example: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

verification object nullable

How to check a request really came from this operator: { method, url }, where method is published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published, and url is the operator's own list (or null). We never fetch it — it is a URL for you to use. Present on curated rows only.

example: {"method":"published-ip-ranges","url":"https://claude.com/crawling/bots.json"}

note string nullable

One curated caveat in our own words, where the operator's documentation says something that changes how you use the token. Curated rows only.

example: Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity.

addedOn string (YYYY-MM-DD) nullable

Date the pattern entered the community list — NOT the date the crawler began operating. null on 55 rows.

example: 2015-04-14

Documented examples

Build-generated requests and complete responses 5

Every AI crawler in the list /api/crawlers?tag=ai-crawler&count=100

GET /api/crawlers?tag=ai-crawler&count=100

{
  "data": [
    {
      "pattern": "AI2Bot",
      "patternSource": "crawler-user-agents",
      "tags": [
        "ai-crawler",
        "academic"
      ],
      "operator": null,
      "operatorId": null,
      "robotsToken": null,
      "purpose": null,
      "exampleUserAgent": "AI2Bot",
      "referenceUrl": "https://datadome.co/bots/ai2bot/",
      "operatorDocsUrl": null,
      "verification": null,
      "note": null,
      "addedOn": "2026-04-17"
    },
    {
      "pattern": "AISearchBot",
      "patternSource": "crawler-user-agents",
      "tags": [
        "search-engine",
        "ai-crawler"
      ],
      "operator": null,
      "operatorId": null,
      "robotsToken": null,
      "purpose": null,
      "exampleUserAgent": null,
      "referenceUrl": null,
      "operatorDocsUrl": null,
      "verification": null,
      "note": null,
      "addedOn": null
    },
    {
      "pattern": "Amzn-SearchBot",
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

Only the bots their operators document themselves /api/crawlers?documented=true

GET /api/crawlers?documented=true

{
  "data": [
    {
      "pattern": "Amazonbot",
      "patternSource": "crawler-user-agents",
      "tags": [
        "search-engine"
      ],
      "operator": "Amazon",
      "operatorId": "amazon",
      "robotsToken": "Amazonbot",
      "purpose": "general",
      "exampleUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)",
      "referenceUrl": "https://developer.amazon.com/support/amazonbot",
      "operatorDocsUrl": "https://developer.amazon.com/amazonbot",
      "verification": {
        "method": "published-ip-ranges",
        "url": "https://developer.amazon.com/amazonbot/ip-addresses/"
      },
      "note": "Amazon states this crawler \"is used to improve our products and services\" and that the data \"may be used to train Amazon AI models\" — the stated purpose is generic and training is named as a further possible use, so purpose stays 'general' and the training sentence stays here rather than being promoted to 'ai-training'. Amazon also states its crawlers do not support the crawl-delay directive. Chrome/W.X.Y.Z is Amazon's own placeholder.",
      "addedOn": "2020-03-02"
    },
    {
      "pattern": "Amzn-SearchBot",
      "patternSource": "crawler-user-agents",
      "tags": [
        "ai-crawler"
      ],
      "operator": "Amazon",
      "operatorId": "amazon",
      "robotsToken": "Amzn-SearchBot",
      "purpose": "search",
      "exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36",
      "referenceUrl": "https://developer.amazon.com/amazonbot",
      "operatorDocsUrl": "https://developer.amazon.com/amazonbot",
      "verification": {
        "method": "published-ip-ranges",
        "url": "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/"
      },
      "note": "Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

Everything Perplexity runs /api/crawlers?operator=perplexity

GET /api/crawlers?operator=perplexity

{
  "data": [
    {
      "pattern": "Perplexity-User",
      "patternSource": "crawler-user-agents",
      "tags": [
        "search-engine",
        "ai-crawler"
      ],
      "operator": "Perplexity",
      "operatorId": "perplexity",
      "robotsToken": "Perplexity-User",
      "purpose": "user-triggered",
      "exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",
      "referenceUrl": "https://docs.perplexity.ai/guides/bots",
      "operatorDocsUrl": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
      "verification": {
        "method": "published-ip-ranges",
        "url": "https://www.perplexity.com/perplexity-user.json"
      },
      "note": "Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules.",
      "addedOn": "2026-04-07"
    },
    {
      "pattern": "PerplexityBot\\/",
      "patternSource": "crawler-user-agents",
      "tags": [
        "search-engine",
        "ai-crawler"
      ],
      "operator": "Perplexity",
      "operatorId": "perplexity",
      "robotsToken": "PerplexityBot",
      "purpose": "search",
      "exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
      "referenceUrl": "https://docs.perplexity.ai/docs/perplexitybot",
      "operatorDocsUrl": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
      "verification": {
        "method": "published-ip-ranges",
        "url": "https://www.perplexity.com/perplexitybot.json"
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

Second page of the SEO crawlers /api/crawlers?tag=seo&count=50&offset=50

GET /api/crawlers?tag=seo&count=50&offset=50

{
  "data": [
    {
      "pattern": "Caliperbot",
      "patternSource": "crawler-user-agents",
      "tags": [
        "seo"
      ],
      "operator": null,
      "operatorId": null,
      "robotsToken": null,
      "purpose": null,
      "exampleUserAgent": null,
      "referenceUrl": "http://www.conductor.com/caliperbot",
      "operatorDocsUrl": null,
      "verification": null,
      "note": null,
      "addedOn": "2019-03-02"
    },
    {
      "pattern": "CapitalOneShopping",
      "patternSource": "crawler-user-agents",
      "tags": [
        "seo"
      ],
      "operator": null,
      "operatorId": null,
      "robotsToken": null,
      "purpose": null,
      "exampleUserAgent": null,
      "referenceUrl": "https://datadome.co/bots/capital-one-shopping-bot/",
      "operatorDocsUrl": null,
      "verification": null,
      "note": null,
      "addedOn": "2026-04-17"
    },
    {
      "pattern": "CentComBot\\/",
      "patternSource": "crawler-user-agents",
      "tags": [
    …
    ]
  }
}

The whole AI-crawler page as CSV /api/crawlers?tag=ai-crawler&count=100&format=csv

GET /api/crawlers?tag=ai-crawler&count=100&format=csv

pattern,patternSource,tags,operator,operatorId,robotsToken,purpose,exampleUserAgent,referenceUrl,operatorDocsUrl,verification,note,addedOn,verification.method,verification.url
AI2Bot,crawler-user-agents,"[""ai-crawler"",""academic""]",,,,,AI2Bot,https://datadome.co/bots/ai2bot/,,,,2026-04-17,,
AISearchBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,,,,,,,,
Amzn-SearchBot,crawler-user-agents,"[""ai-crawler""]",Amazon,amazon,Amzn-SearchBot,search,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36",https://developer.amazon.com/amazonbot,https://developer.amazon.com/amazonbot,,"Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",2026-04-26,published-ip-ranges,https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Amzn-User,crawler-user-agents,"[""ai-crawler""]",Amazon,amazon,Amzn-User,user-triggered,Mozilla/5.0 (compatible; Amzn-User; +https://developer.amazon.com/),https://developer.amazon.com/amazonbot,https://developer.amazon.com/amazonbot,,Amazon states this agent fetches live information from the web on a user's behalf and does not crawl content for generative AI model training.,2026-04-26,published-ip-ranges,https://developer.amazon.com/amazonbot/live-ip-addresses/
Anomura,crawler-user-agents,"[""ai-crawler""]",,,,,Anomura/1.2 (+https://www.direqt.ai),https://docs.direqt-search.com/direqt-bots/direqt-crawlers-and-user-agents,,,,2026-04-26,,
ApifyBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ApifyBot/1.0),https://knownagents.com/agents/apifybot,,,,2026-04-26,,
ApifyWebsiteContentCrawler,crawler-user-agents,"[""ai-crawler""]",,,,,ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler),https://apify.com/apify/website-content-crawler,,,,2026-04-26,,
Aranet-SearchBot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36",https://aranet.ai/bot,,,,2026-04-26,,
AzureAI-SearchBot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;",https://azure.microsoft.com/en-us/products/ai-services,,,,2026-04-26,,
Brightbot,crawler-user-agents,"[""ai-crawler""]",,,,,Brightbot 1.0,https://brightdata.com/brightbot,,,,2026-04-26,,
Bytespider,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com",https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent,,,,2019-11-11,,
CCBot,crawler-user-agents,"[""ai-crawler""]",Common Crawl,common-crawl,CCBot,ai-training,CCBot/2.0 (https://commoncrawl.org/faq/),http://www.commoncrawl.org/bot.html,https://commoncrawl.org/ccbot,,"Common Crawl's page describes CCBot only as producing ""an open repository of web crawl data that is universally accessible and analyzable by anyone"" — it does not mention AI training at all, so purpose 'ai-training' here is our classification, not Common Crawl's wording: the corpus is the canonical public training corpus, and anyone disallowing training crawlers needs this token. The page also warns that other crawlers falsely identify themselves as CCBot, and that real CCBot traffic reverse-resolves under crawl.commoncrawl.org (IPv4 only).",2012-02-05,reverse-dns-and-ip-ranges,https://index.commoncrawl.org/ccbot.json
Channel3Bot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Channel3Bot/1.0; +https://trychannel3.com/channel3bot),https://trychannel3.com/channel3bot,,,,2026-04-26,,
ChatGLM-Spider,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/),https://chatglm.cn/,,,,2026-04-17,,
ChatGPT-User,crawler-user-agents,"[""ai-crawler""]",OpenAI,openai,ChatGPT-User,user-triggered,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",https://openai.com/bot,https://developers.openai.com/api/docs/bots,,"OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",2024-04-19,published-ip-ranges,https://openai.com/chatgpt-user.json
Claude-SearchBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",Anthropic,anthropic,Claude-SearchBot,search,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)",https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,,"Anthropic states this bot analyses content to improve search result quality, and that disabling it prevents indexing for search and may reduce a site's visibility in search results.",2026-04-07,published-ip-ranges,https://claude.com/crawling/bots.json
Claude-User,crawler-user-agents,"[""ai-crawler""]",Anthropic,anthropic,Claude-User,user-triggered,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)",https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,,"Anthropic states this agent fetches pages when a person asks Claude a question, and lists it among the bots that honour robots.txt — unlike several other operators' user-triggered fetchers.",2026-04-07,published-ip-ranges,https://claude.com/crawling/bots.json
Claude-Web,crawler-user-agents,"[""ai-crawler""]",,,,,Claude-Web/1.0 (web crawler; +https://www.anthropic.com/; bots@anthropic.com),https://anthropic.com,,,,2026-04-07,,
Cloudflare-AutoRAG,crawler-user-agents,"[""ai-crawler""]",,,,,Cloudflare-AutoRAG (https://developers.cloudflare.com/autorag; autorag@cloudflare.com),https://developers.cloudflare.com/autorag,,,,2026-04-26,,
DeepSeekBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot),http://deepseek.com/bot,,,,2026-04-26,,
Devin,crawler-user-agents,"[""ai-crawler"",""browser-automation""]",,,,,"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36; Devin/1.0; +https://devin.ai",https://docs.devin.ai/get-started/devin-intro,,,,2026-04-17,,
DuckAssistBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html),https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot,,,,2026-04-07,,
ExteContextCrawl,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ExteContextCrawl/1.0; +http://crawl001.exte.ai),http://crawl001.exte.ai/,,,,2026-04-26,,
FacebookBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/facebookbot/),https://developers.facebook.com/docs/sharing/bot/,,,,2026-04-26,,
FirecrawlAgent,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; FirecrawlAgent; +https://firecrawl.dev/),https://firecrawl.dev/,,,,2026-04-26,,
Flyriverbot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Flyriverbot/1.1 (+https://www.flyriver.com/; AI Content Source Check)",https://flyriver.com/crawler,,,,2026-04-26,,
GPTBot,crawler-user-agents,"[""ai-crawler""]",OpenAI,openai,GPTBot,ai-training,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)",https://platform.openai.com/docs/gptbot,https://developers.openai.com/api/docs/bots,,OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.,2023-08-09,published-ip-ranges,https://openai.com/gptbot.json
Gemini-Deep-Research,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36",https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers,,,,2026-04-07,,
Google-CloudVertexBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",Google,google,Google-CloudVertexBot,site-owner-requested,,https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-cloudvertexbot,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,,"Google states preferences for this token affect crawls that site owners themselves request for building Vertex AI Agents, and have no effect on Google Search. Google publishes only the user-agent substring for it, not a full string, so fullUserAgentExample is null.",2026-04-17,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Google-Extended,crawler-user-agents,"[""ai-crawler""]",Google,google,Google-Extended,ai-training,,https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,,"Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect a site's inclusion in Google Search nor act as a ranking signal. Google also states the token governs two distinct things, not just one: whether crawled content may be used for training future generations of the Gemini models behind Gemini Apps and the Vertex AI API, AND grounding — supplying content from the Google Search index to the model at prompt time — in Gemini Apps and Grounding with Google Search on Vertex AI. Disallowing it turns off both. Nothing ever sends this string, so the verification link is for Google's actual crawler traffic.",2026-04-07,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Google-NotebookLM,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Google-NotebookLM; +https://notebooklm.google.com/),https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers,,,,2026-04-26,,
HenkBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; HenkBot/1.0; +https://valyu.ai/crawler),https://valyu.ai/crawler,,,,2026-04-26,,
ImageMind,crawler-user-agents,"[""ai-crawler""]",,,,,ImageMind,https://datadome.co/bots/imagemind/,,,,2026-04-17,,
Kangaroo Bot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Kangaroo Bot/1.0; +http://www.kangaroo.com),https://kangaroollm.com.au/kangaroo-bot/,,,,2026-04-26,,
KendraBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,KendraBot,https://docs.aws.amazon.com/kendra/latest/dg/what-is-kendra.html,,,,2026-04-17,,
KunatoCrawler,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; KunatoCrawler/1.0; +http://kunato.ai/bot.html),http://kunato.ai/bot.html,,,,2026-04-26,,
LinerBot\/,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinerBot/1.0; +https://docs.getliner.com/docs/linerbot)",https://docs.getliner.com/docs/linerbot,,,,2026-04-17,,
LinkupBot,crawler-user-agents,"[""ai-crawler""]",,,,,LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so),https://linkup.so/bot,,,,2026-04-26,,
MistralAI-User,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)",,,,,2026-04-07,,
    …
newsai\/,crawler-user-agents,"[""feed-reader"",""ai-crawler""]",,,,,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36",https://knownagents.com/agents/newsai,,,,2026-04-14,,
semantic-visions,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (Linux; CentOS; compatible; semantic-visions-discovery; HTTPClient 4.5),https://semantic-visions.com/,,,,2026-04-26,,

GET /api/crawlers/detect Check a user-agent string against every known crawler pattern Computed answer

Scans the string you send against all 1500 bundled patterns and returns every match. Pure function of 'userAgent': no clock, no network, no state. The string is taken byte for byte — nothing is trimmed, because 13 patterns are anchored and a stray leading space genuinely changes the verdict.

Live requestRuns against the public API

No key required

GET

/api/crawlers/detect?userAgent=Mozilla%2F5.0+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29%3B+compatible%3B+GPTBot%2F1.4%3B+%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot

Request parameters

userAgentrequired

The User-Agent header value to test, exactly as received. Taken byte for byte: leading and trailing whitespace is kept (and warned about) rather than stripped, matching is case-sensitive, and a CR or LF is a 400 because an HTTP field value is a single line. Capped at 1024 characters, which every string bundled here fits inside.

Advanced response options4 options

fields

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

exclude

Return all fields except these (comma-separated).

prettydefault false

Pretty-print the JSON response.

unwrapdefault false

Drop the envelope: return the raw array/object without data/meta wrapper.

Response

Example parameters are ready. Send the request to inspect the live response.

Route reference

Code, request parameters, response fields and verified examples 5 params · 6 fields · 4 examples

Code samples Ready-to-copy requests in 4 languages

curl JavaScript Python PHP

curl "https://randomapi.dev/api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot"

const res = await fetch("https://randomapi.dev/api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot");
const { data, meta } = await res.json();

import requests

data = requests.get("https://randomapi.dev/api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot").json()["data"]

$json = json_decode(file_get_contents(
  "https://randomapi.dev/api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot"
), true);
$data = $json["data"];

Parameters Route-specific request inputs 1

Parameter	Type	Default & allowed	Description
`userAgent` required	string	example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot	The User-Agent header value to test, exactly as received. Taken byte for byte: leading and trailing whitespace is kept (and warned about) rather than stripped, matching is case-sensitive, and a CR or LF is a 400 because an HTTP field value is a single line. Capped at 1024 characters, which every string bundled here fits inside.

userAgent string required

example: userAgent=Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Universal parameters Shared response and formatting options 4

Parameter	Type	Default & allowed	Description
`fields`	list	example: userAgent,isKnownCrawler	Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
`exclude`	list	example: robotsTokens	Return all fields except these (comma-separated).
`pretty`	boolean	default: false example: true	Pretty-print the JSON response.
`unwrap`	boolean	default: false example: true	Drop the envelope: return the raw array/object without data/meta wrapper.

fields list

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

example: fields=userAgent,isKnownCrawler

exclude list

Return all fields except these (comma-separated).

example: exclude=robotsTokens

pretty boolean

Pretty-print the JSON response.

default: false

example: pretty=true

unwrap boolean

Drop the envelope: return the raw array/object without data/meta wrapper.

default: false

example: unwrap=true

Response schema Fields returned in each record 6

Field	Type	Description	Example
`userAgent`	string	The string that was tested, echoed exactly as supplied — nothing is trimmed.	CCBot/2.0 (https://commoncrawl.org/faq/)
`isKnownCrawler`	boolean	Whether at least one bundled pattern matched. false means 'no pattern in this snapshot matched' — it never means the request came from a human. Anyone can send any User-Agent, and crawlers that disguise themselves as browsers are exactly the ones a string list cannot see.	true
`matchCount`	integer	Exact number of patterns that matched, even when matches is capped at 25.	1
`matches`	object[]	Every matching row in the same 13-field shape as /api/crawlers, ordered curated-first, then longest pattern first, then pattern ascending. Capped at 25 entries with a warning; matchCount stays exact. Empty array when nothing matched.	[{"pattern":"CCBot","robotsToken":"CCBot","operator":"Common Crawl"}]
`operators`	string[]	Distinct curated operators among the returned matches, in first-appearance order. Empty when no matched row is curated.	["Common Crawl"]
`robotsTokens`	string[]	Distinct curated robotsToken values among the returned matches — the lines to write in robots.txt if you want to control this traffic. Empty when nothing curated matched.	["CCBot"]

userAgent string

The string that was tested, echoed exactly as supplied — nothing is trimmed.

example: CCBot/2.0 (https://commoncrawl.org/faq/)

isKnownCrawler boolean

Whether at least one bundled pattern matched. false means 'no pattern in this snapshot matched' — it never means the request came from a human. Anyone can send any User-Agent, and crawlers that disguise themselves as browsers are exactly the ones a string list cannot see.

example: true

matchCount integer

Exact number of patterns that matched, even when matches is capped at 25.

example: 1

matches object[]

Every matching row in the same 13-field shape as /api/crawlers, ordered curated-first, then longest pattern first, then pattern ascending. Capped at 25 entries with a warning; matchCount stays exact. Empty array when nothing matched.

example: [{"pattern":"CCBot","robotsToken":"CCBot","operator":"Common Crawl"}]

operators string[]

Distinct curated operators among the returned matches, in first-appearance order. Empty when no matched row is curated.

example: ["Common Crawl"]

robotsTokens string[]

Distinct curated robotsToken values among the returned matches — the lines to write in robots.txt if you want to control this traffic. Empty when nothing curated matched.

example: ["CCBot"]

Documented examples

Build-generated requests and complete responses 4

A real GPTBot request

 /api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot

GET /api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot

{
  "data": {
    "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
    "isKnownCrawler": true,
    "matchCount": 1,
    "matches": [
      {
        "pattern": "GPTBot",
        "patternSource": "crawler-user-agents",
        "tags": [
          "ai-crawler"
        ],
        "operator": "OpenAI",
        "operatorId": "openai",
        "robotsToken": "GPTBot",
        "purpose": "ai-training",
        "exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)",
        "referenceUrl": "https://platform.openai.com/docs/gptbot",
        "operatorDocsUrl": "https://developers.openai.com/api/docs/bots",
        "verification": {
          "method": "published-ip-ranges",
          "url": "https://openai.com/gptbot.json"
        },
        "note": "OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.",
        "addedOn": "2023-08-09"
      }
    ],
    "operators": [
      "OpenAI"
    ],
    "robotsTokens": [
      "GPTBot"
    ]
  },
  "meta": {
    "endpoint": "crawlers",
    "route": "/detect",
    "params": {
      "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
    },
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

An ordinary Chrome desktop browser

 /api/crawlers/detect?userAgent=Mozilla%2F5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F138.0.0.0%20Safari%2F537.36

GET /api/crawlers/detect?userAgent=Mozilla%2F5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F138.0.0.0%20Safari%2F537.36

{
  "data": {
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    "isKnownCrawler": false,
    "matchCount": 0,
    "matches": [],
    "operators": [],
    "robotsTokens": []
  },
  "meta": {
    "endpoint": "crawlers",
    "route": "/detect",
    "params": {
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
    },
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

One string, two matching patterns

 /api/crawlers/detect?userAgent=linkdexbot%2FNutch-1.0-dev%20(http%3A%2F%2Fwww.linkdex.com%2F%3B%20crawl%20at%20linkdex%20dot%20com)

GET /api/crawlers/detect?userAgent=linkdexbot%2FNutch-1.0-dev%20(http%3A%2F%2Fwww.linkdex.com%2F%3B%20crawl%20at%20linkdex%20dot%20com)

{
  "data": {
    "userAgent": "linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)",
    "isKnownCrawler": true,
    "matchCount": 2,
    "matches": [
      {
        "pattern": "linkdex",
        "patternSource": "crawler-user-agents",
        "tags": [
          "seo"
        ],
        "operator": null,
        "operatorId": null,
        "robotsToken": null,
        "purpose": null,
        "exampleUserAgent": null,
        "referenceUrl": "http://www.linkdex.com",
        "operatorDocsUrl": null,
        "verification": null,
        "note": null,
        "addedOn": "2011-01-06"
      },
      {
        "pattern": "Nutch",
        "patternSource": "crawler-user-agents",
        "tags": [
          "search-engine"
        ],
        "operator": null,
        "operatorId": null,
        "robotsToken": null,
        "purpose": null,
        "exampleUserAgent": null,
        "referenceUrl": null,
        "operatorDocsUrl": null,
        "verification": null,
        "note": null,
        "addedOn": null
      }
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

Case matters /api/crawlers/detect?userAgent=googlebot%2F2.1

GET /api/crawlers/detect?userAgent=googlebot%2F2.1

{
  "data": {
    "userAgent": "googlebot/2.1",
    "isKnownCrawler": false,
    "matchCount": 0,
    "matches": [],
    "operators": [],
    "robotsTokens": []
  },
  "meta": {
    "endpoint": "crawlers",
    "route": "/detect",
    "params": {
      "userAgent": "googlebot/2.1"
    },
    "generatedAt": "2026-07-30T15:14:08.000Z",
    "warnings": [
      "No pattern matched, but 1 would match if case were ignored: 'Googlebot\\/'. The bundled patterns are matched case-sensitively because they encode their own case alternatives (for example '[cC]laude[bB]ot'), so a lowercased log line does not match — check the capitalisation of the string you sent."
    ]
  }
}

GET /api/crawlers/robots-tokens The robots.txt tokens each operator publishes for its own crawlers Returns a list

The curated half of this endpoint: 24 tokens from 8 operators, each read off that operator's own documentation page and last checked on 2026-07-30. A curated subset per operator, not a complete catalogue — follow each row's 'docsUrl' for the operator's full list. Ordered by operatorId then token.

Live requestRuns against the public API

No key required

GET/api/crawlers/robots-tokens?operator=openai

Request parameters

operatordefault any

Only this operator's tokens.

purposedefault any

Only tokens recorded with this purpose. The value restates the operator's own published wording on every row except CCBot and meta-webindexer, where the row's note says the label is ours.

robotsTxtBehaviordefault any

Only tokens whose operator publishes this robots.txt behaviour about itself. 'not-published' selects the tokens whose page says nothing either way.

Advanced response options6 options

countdefault all matches

Limit the number of returned records (1–100). Defaults to all matches.

fields

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

exclude

Return all fields except these (comma-separated).

formatdefault json

Response format: json envelope, ndjson (one record per line) or csv.

prettydefault false

Pretty-print the JSON response.

unwrapdefault false

Drop the envelope: return the raw array/object without data/meta wrapper.

Response

Example parameters are ready. Send the request to inspect the live response.

Route reference

Code, request parameters, response fields and verified examples 9 params · 15 fields · 4 examples

Code samples Ready-to-copy requests in 4 languages

curl JavaScript Python PHP

curl "https://randomapi.dev/api/crawlers/robots-tokens?operator=openai"

const res = await fetch("https://randomapi.dev/api/crawlers/robots-tokens?operator=openai");
const { data, meta } = await res.json();

import requests

data = requests.get("https://randomapi.dev/api/crawlers/robots-tokens?operator=openai").json()["data"]

$json = json_decode(file_get_contents(
  "https://randomapi.dev/api/crawlers/robots-tokens?operator=openai"
), true);
$data = $json["data"];

Parameters Route-specific request inputs 3

Parameter	Type	Default & allowed	Description
`operator`	enum	default: any allowed: openai \| anthropic \| perplexity \| google \| apple \| common-crawl \| meta \| amazon \| any example: openai	Only this operator's tokens.
`purpose`	enum	default: any allowed: search \| ai-training \| user-triggered \| advertising \| site-owner-requested \| link-preview \| general \| any example: ai-training	Only tokens recorded with this purpose. The value restates the operator's own published wording on every row except CCBot and meta-webindexer, where the row's note says the label is ours.
`robotsTxtBehavior`	enum	default: any allowed: obeys \| obeys-except-integrity-checks \| user-triggered-may-ignore \| not-published \| any example: user-triggered-may-ignore	Only tokens whose operator publishes this robots.txt behaviour about itself. 'not-published' selects the tokens whose page says nothing either way.

operator enum

Only this operator's tokens.

default: any

example: operator=openai

purpose enum

Only tokens recorded with this purpose. The value restates the operator's own published wording on every row except CCBot and meta-webindexer, where the row's note says the label is ours.

default: any

example: purpose=ai-training

robotsTxtBehavior enum

Only tokens whose operator publishes this robots.txt behaviour about itself. 'not-published' selects the tokens whose page says nothing either way.

default: any

allowed: obeys | obeys-except-integrity-checks | user-triggered-may-ignore | not-published | any

example: robotsTxtBehavior=user-triggered-may-ignore

Universal parameters Shared response and formatting options 6

Parameter	Type	Default & allowed	Description
`count`	int	default: all matches allowed: 1 – 100 example: 3	Limit the number of returned records (1–100). Defaults to all matches.
`fields`	list	example: operatorId,operator	Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
`exclude`	list	example: verifiedOn	Return all fields except these (comma-separated).
`format`	enum	default: json allowed: json \| ndjson \| csv example: csv	Response format: json envelope, ndjson (one record per line) or csv.
`pretty`	boolean	default: false example: true	Pretty-print the JSON response.
`unwrap`	boolean	default: false example: true	Drop the envelope: return the raw array/object without data/meta wrapper.

count int

Limit the number of returned records (1–100). Defaults to all matches.

default: all matches

allowed: 1 – 100

example: count=3

fields list

Return only these fields (comma-separated). Mutually exclusive with 'exclude'.

example: fields=operatorId,operator

exclude list

Return all fields except these (comma-separated).

example: exclude=verifiedOn

format enum

Response format: json envelope, ndjson (one record per line) or csv.

default: json

allowed: json | ndjson | csv

example: format=csv

pretty boolean

Pretty-print the JSON response.

default: false

example: pretty=true

unwrap boolean

Drop the envelope: return the raw array/object without data/meta wrapper.

default: false

example: unwrap=true

Response schema Fields returned in each record 15

Field	Type	Description	Example
`operatorId`	string	Filter value for ?operator=.	openai
`operator`	string	Operator's name as it writes it.	OpenAI
`token`	string	The token, in the operator's own spelling. RFC 9309, Section 2.2.1 requires crawlers to match product tokens case-insensitively, so the capitalisation is cosmetic; every token here also satisfies that section's product-token grammar (letters, '_' and '-' only), so no conforming parser truncates it.	GPTBot
`robotsTxtLine`	string	The line to paste: 'User-agent: ' + token. Add your own Disallow: line under it, then test the whole file with /api/robots-txt/check.	User-agent: GPTBot
`purpose`	string	search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. Our restatement of the operator's published wording, never invented — except on the 2 rows whose own note says the label is ours rather than the operator's (today CCBot and meta-webindexer), which is why the note ships next to it.	ai-training
`sendsRequests`	boolean	Whether any request actually carries this string. false for Google-Extended and Applebot-Extended: control tokens only, so you will never see them in an access log.	true
`robotsTxtBehavior`	string	obeys, obeys-except-integrity-checks, user-triggered-may-ignore or not-published, as the operator publishes it about itself. 'not-published' means the operator's page does not say. We do not measure compliance and this is not a claim that any crawler behaves as documented.	obeys
`fullUserAgentExample` nullable	string	The User-Agent string the operator publishes for this token, verbatim — including the operator's own placeholder version tokens (Google and Amazon both publish Chrome/W.X.Y.Z). We never fill a placeholder in. null where the operator publishes none: Anthropic publishes tokens but no strings, Google publishes none for Google-Extended and only a substring for Google-CloudVertexBot.	Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
`pattern`	string	The bundled pattern that detects this token on /api/crawlers/detect, so the two routes are visibly the same universe.	GPTBot
`patternSource`	string	crawler-user-agents when the community list already carries a pattern for this token, operator-documentation for the two tokens where it does not and we authored one from the operator's page (OAI-AdsBot, Applebot-Extended).	crawler-user-agents
`docsUrl`	string	The operator's own documentation page — the citable source for this row.	https://developers.openai.com/api/docs/bots
`verificationMethod`	string	published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published. For the two control-only tokens this describes how to verify the operator's actual crawler traffic, since nothing ever sends the control token itself.	published-ip-ranges
`verificationUrl` nullable	string	The operator's own IP-range list. Never fetched by this API.	https://openai.com/gptbot.json
`note`	string	One caveat in our own words, where the operator's page says something that changes how the token behaves.	Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group.
`verifiedOn`	string (YYYY-MM-DD)	The date this table was last checked against the operators' own pages. A module constant, identical on every row — never the current time.	2026-07-30

operatorId string

Filter value for ?operator=.

example: openai

operator string

Operator's name as it writes it.

example: OpenAI

token string

The token, in the operator's own spelling. RFC 9309, Section 2.2.1 requires crawlers to match product tokens case-insensitively, so the capitalisation is cosmetic; every token here also satisfies that section's product-token grammar (letters, '_' and '-' only), so no conforming parser truncates it.

example: GPTBot

robotsTxtLine string

The line to paste: 'User-agent: ' + token. Add your own Disallow: line under it, then test the whole file with /api/robots-txt/check.

example: User-agent: GPTBot

purpose string

search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. Our restatement of the operator's published wording, never invented — except on the 2 rows whose own note says the label is ours rather than the operator's (today CCBot and meta-webindexer), which is why the note ships next to it.

example: ai-training

sendsRequests boolean

Whether any request actually carries this string. false for Google-Extended and Applebot-Extended: control tokens only, so you will never see them in an access log.

example: true

robotsTxtBehavior string

obeys, obeys-except-integrity-checks, user-triggered-may-ignore or not-published, as the operator publishes it about itself. 'not-published' means the operator's page does not say. We do not measure compliance and this is not a claim that any crawler behaves as documented.

example: obeys

fullUserAgentExample string nullable

The User-Agent string the operator publishes for this token, verbatim — including the operator's own placeholder version tokens (Google and Amazon both publish Chrome/W.X.Y.Z). We never fill a placeholder in. null where the operator publishes none: Anthropic publishes tokens but no strings, Google publishes none for Google-Extended and only a substring for Google-CloudVertexBot.

example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

pattern string

The bundled pattern that detects this token on /api/crawlers/detect, so the two routes are visibly the same universe.

example: GPTBot

patternSource string

crawler-user-agents when the community list already carries a pattern for this token, operator-documentation for the two tokens where it does not and we authored one from the operator's page (OAI-AdsBot, Applebot-Extended).

example: crawler-user-agents

docsUrl string

The operator's own documentation page — the citable source for this row.

example: https://developers.openai.com/api/docs/bots

verificationMethod string

published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published. For the two control-only tokens this describes how to verify the operator's actual crawler traffic, since nothing ever sends the control token itself.

example: published-ip-ranges

verificationUrl string nullable

The operator's own IP-range list. Never fetched by this API.

example: https://openai.com/gptbot.json

note string

One caveat in our own words, where the operator's page says something that changes how the token behaves.

example: Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group.

verifiedOn string (YYYY-MM-DD)

The date this table was last checked against the operators' own pages. A module constant, identical on every row — never the current time.

example: 2026-07-30

Documented examples

Build-generated requests and complete responses 4

Everything OpenAI publishes /api/crawlers/robots-tokens?operator=openai

GET /api/crawlers/robots-tokens?operator=openai

{
  "data": [
    {
      "operatorId": "openai",
      "operator": "OpenAI",
      "token": "ChatGPT-User",
      "robotsTxtLine": "User-agent: ChatGPT-User",
      "purpose": "user-triggered",
      "sendsRequests": true,
      "robotsTxtBehavior": "user-triggered-may-ignore",
      "fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",
      "pattern": "ChatGPT-User",
      "patternSource": "crawler-user-agents",
      "docsUrl": "https://developers.openai.com/api/docs/bots",
      "verificationMethod": "published-ip-ranges",
      "verificationUrl": "https://openai.com/chatgpt-user.json",
      "note": "OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "openai",
      "operator": "OpenAI",
      "token": "GPTBot",
      "robotsTxtLine": "User-agent: GPTBot",
      "purpose": "ai-training",
      "sendsRequests": true,
      "robotsTxtBehavior": "obeys",
      "fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
      "pattern": "GPTBot",
      "patternSource": "crawler-user-agents",
      "docsUrl": "https://developers.openai.com/api/docs/bots",
      "verificationMethod": "published-ip-ranges",
      "verificationUrl": "https://openai.com/gptbot.json",
      "note": "OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "openai",
      "operator": "OpenAI",
      "token": "OAI-AdsBot",
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

The training bots, across every operator /api/crawlers/robots-tokens?purpose=ai-training

GET /api/crawlers/robots-tokens?purpose=ai-training

{
  "data": [
    {
      "operatorId": "anthropic",
      "operator": "Anthropic",
      "token": "ClaudeBot",
      "robotsTxtLine": "User-agent: ClaudeBot",
      "purpose": "ai-training",
      "sendsRequests": true,
      "robotsTxtBehavior": "obeys",
      "fullUserAgentExample": null,
      "pattern": "[cC]laude[bB]ot",
      "patternSource": "crawler-user-agents",
      "docsUrl": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
      "verificationMethod": "published-ip-ranges",
      "verificationUrl": "https://claude.com/crawling/bots.json",
      "note": "Anthropic states that restricting ClaudeBot signals that a site's future material should be excluded from its AI model training datasets, and that its bots honour robots.txt and the non-standard Crawl-delay extension. Anthropic publishes tokens but no user-agent strings.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "apple",
      "operator": "Apple",
      "token": "Applebot-Extended",
      "robotsTxtLine": "User-agent: Applebot-Extended",
      "purpose": "ai-training",
      "sendsRequests": false,
      "robotsTxtBehavior": "obeys",
      "fullUserAgentExample": null,
      "pattern": "Applebot-Extended",
      "patternSource": "operator-documentation",
      "docsUrl": "https://support.apple.com/en-us/119829",
      "verificationMethod": "reverse-dns-and-ip-ranges",
      "verificationUrl": "https://search.developer.apple.com/applebot.json",
      "note": "Opt-out control only: Apple states Applebot-Extended does not crawl webpages, that pages disallowing it can still appear in search results, and that it only determines how data already crawled by Applebot may be used to train Apple's foundation models. The community snapshot carries no pattern for it, so this row's pattern was authored from Apple's page.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "common-crawl",
      "operator": "Common Crawl",
      "token": "CCBot",
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

The fetchers that say they may ignore robots.txt /api/crawlers/robots-tokens?robotsTxtBehavior=user-triggered-may-ignore

GET /api/crawlers/robots-tokens?robotsTxtBehavior=user-triggered-may-ignore

{
  "data": [
    {
      "operatorId": "meta",
      "operator": "Meta",
      "token": "meta-externalfetcher",
      "robotsTxtLine": "User-agent: meta-externalfetcher",
      "purpose": "user-triggered",
      "sendsRequests": true,
      "robotsTxtBehavior": "user-triggered-may-ignore",
      "fullUserAgentExample": "meta-externalfetcher/1.1",
      "pattern": "meta-externalfetcher\\/",
      "patternSource": "crawler-user-agents",
      "docsUrl": "https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers",
      "verificationMethod": "none-published",
      "verificationUrl": null,
      "note": "Meta states this crawler fetches individual links at a user's request, supports agentic AI capabilities, and accordingly may bypass robots.txt rules.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "openai",
      "operator": "OpenAI",
      "token": "ChatGPT-User",
      "robotsTxtLine": "User-agent: ChatGPT-User",
      "purpose": "user-triggered",
      "sendsRequests": true,
      "robotsTxtBehavior": "user-triggered-may-ignore",
      "fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",
      "pattern": "ChatGPT-User",
      "patternSource": "crawler-user-agents",
      "docsUrl": "https://developers.openai.com/api/docs/bots",
      "verificationMethod": "published-ip-ranges",
      "verificationUrl": "https://openai.com/chatgpt-user.json",
      "note": "OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",
      "verifiedOn": "2026-07-30"
    },
    {
      "operatorId": "perplexity",
      "operator": "Perplexity",
      "token": "Perplexity-User",
    …
    "generatedAt": "2026-07-30T15:14:08.000Z"
  }
}

The whole table as CSV /api/crawlers/robots-tokens?format=csv

GET /api/crawlers/robots-tokens?format=csv

operatorId,operator,token,robotsTxtLine,purpose,sendsRequests,robotsTxtBehavior,fullUserAgentExample,pattern,patternSource,docsUrl,verificationMethod,verificationUrl,note,verifiedOn
amazon,Amazon,Amazonbot,User-agent: Amazonbot,general,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36",Amazonbot,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/ip-addresses/,"Amazon states this crawler ""is used to improve our products and services"" and that the data ""may be used to train Amazon AI models"" — the stated purpose is generic and training is named as a further possible use, so purpose stays 'general' and the training sentence stays here rather than being promoted to 'ai-training'. Amazon also states its crawlers do not support the crawl-delay directive. Chrome/W.X.Y.Z is Amazon's own placeholder.",2026-07-30
amazon,Amazon,Amzn-SearchBot,User-agent: Amzn-SearchBot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36",Amzn-SearchBot,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/searchbot-ip-addresses/,"Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",2026-07-30
amazon,Amazon,Amzn-User,User-agent: Amzn-User,user-triggered,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36",Amzn-User,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/live-ip-addresses/,Amazon states this agent fetches live information from the web on a user's behalf and does not crawl content for generative AI model training.,2026-07-30
anthropic,Anthropic,Claude-SearchBot,User-agent: Claude-SearchBot,search,true,obeys,,Claude-SearchBot,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states this bot analyses content to improve search result quality, and that disabling it prevents indexing for search and may reduce a site's visibility in search results.",2026-07-30
anthropic,Anthropic,Claude-User,User-agent: Claude-User,user-triggered,true,obeys,,Claude-User,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states this agent fetches pages when a person asks Claude a question, and lists it among the bots that honour robots.txt — unlike several other operators' user-triggered fetchers.",2026-07-30
anthropic,Anthropic,ClaudeBot,User-agent: ClaudeBot,ai-training,true,obeys,,[cC]laude[bB]ot,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states that restricting ClaudeBot signals that a site's future material should be excluded from its AI model training datasets, and that its bots honour robots.txt and the non-standard Crawl-delay extension. Anthropic publishes tokens but no user-agent strings.",2026-07-30
apple,Apple,Applebot,User-agent: Applebot,search,true,obeys,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)",Applebot,crawler-user-agents,https://support.apple.com/en-us/119829,reverse-dns-and-ip-ranges,https://search.developer.apple.com/applebot.json,"Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group, that Applebot does not follow crawl-delay, and that crawled data may also help train Apple's foundation models unless Applebot-Extended is disallowed. Traffic reverse-resolves under *.applebot.apple.com.",2026-07-30
apple,Apple,Applebot-Extended,User-agent: Applebot-Extended,ai-training,false,obeys,,Applebot-Extended,operator-documentation,https://support.apple.com/en-us/119829,reverse-dns-and-ip-ranges,https://search.developer.apple.com/applebot.json,"Opt-out control only: Apple states Applebot-Extended does not crawl webpages, that pages disallowing it can still appear in search results, and that it only determines how data already crawled by Applebot may be used to train Apple's foundation models. The community snapshot carries no pattern for it, so this row's pattern was authored from Apple's page.",2026-07-30
common-crawl,Common Crawl,CCBot,User-agent: CCBot,ai-training,true,obeys,CCBot/2.0 (https://commoncrawl.org/faq/),CCBot,crawler-user-agents,https://commoncrawl.org/ccbot,reverse-dns-and-ip-ranges,https://index.commoncrawl.org/ccbot.json,"Common Crawl's page describes CCBot only as producing ""an open repository of web crawl data that is universally accessible and analyzable by anyone"" — it does not mention AI training at all, so purpose 'ai-training' here is our classification, not Common Crawl's wording: the corpus is the canonical public training corpus, and anyone disallowing training crawlers needs this token. The page also warns that other crawlers falsely identify themselves as CCBot, and that real CCBot traffic reverse-resolves under crawl.commoncrawl.org (IPv4 only).",2026-07-30
google,Google,Google-CloudVertexBot,User-agent: Google-CloudVertexBot,site-owner-requested,true,obeys,,Google-CloudVertexBot,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google states preferences for this token affect crawls that site owners themselves request for building Vertex AI Agents, and have no effect on Google Search. Google publishes only the user-agent substring for it, not a full string, so fullUserAgentExample is null.",2026-07-30
google,Google,Google-Extended,User-agent: Google-Extended,ai-training,false,obeys,,Google-Extended,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect a site's inclusion in Google Search nor act as a ranking signal. Google also states the token governs two distinct things, not just one: whether crawled content may be used for training future generations of the Gemini models behind Gemini Apps and the Vertex AI API, AND grounding — supplying content from the Google Search index to the model at prompt time — in Gemini Apps and Grounding with Google Search on Vertex AI. Disallowing it turns off both. Nothing ever sends this string, so the verification link is for Google's actual crawler traffic.",2026-07-30
google,Google,GoogleOther,User-agent: GoogleOther,general,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36",GoogleOther,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google describes GoogleOther as the generic crawler that various product teams may use, and says crawling preferences for it do not affect any specific product.",2026-07-30
google,Google,Googlebot,User-agent: Googlebot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36",Googlebot\/,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google publishes several Googlebot strings (this desktop one, a smartphone one and two shorter legacy forms); Chrome/W.X.Y.Z is Google's own version placeholder and is kept verbatim. Google says its common crawlers reverse-resolve under crawl-*.googlebot.com.",2026-07-30
meta,Meta,facebookexternalhit,User-agent: facebookexternalhit,link-preview,true,obeys-except-integrity-checks,facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php),facebookexternalhit,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content. Meta's page publishes no IP ranges or reverse-DNS domain, so there is no first-party verification link for any Meta crawler.",2026-07-30
meta,Meta,meta-externalads,User-agent: meta-externalads,advertising,true,obeys,meta-externalads/1.1,meta-externalads\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,Meta states this crawler supports use cases such as improving advertising and other business-related products.,2026-07-30
meta,Meta,meta-externalagent,User-agent: meta-externalagent,ai-training,true,obeys,meta-externalagent/1.1,meta-externalagent\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,Meta states this crawler is used for purposes such as training foundation AI models or improving products.,2026-07-30
meta,Meta,meta-externalfetcher,User-agent: meta-externalfetcher,user-triggered,true,user-triggered-may-ignore,meta-externalfetcher/1.1,meta-externalfetcher\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler fetches individual links at a user's request, supports agentic AI capabilities, and accordingly may bypass robots.txt rules.",2026-07-30
meta,Meta,meta-webindexer,User-agent: meta-webindexer,search,true,obeys,meta-webindexer/1.1,meta-webindexer,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler ""navigates the web to improve Meta AI search result quality for users"" and that allowing it helps Meta cite and link the site in Meta AI's responses; we record that as purpose 'search', which is our classification of Meta's wording rather than a quote from it.",2026-07-30
openai,OpenAI,ChatGPT-User,User-agent: ChatGPT-User,user-triggered,true,user-triggered-may-ignore,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",ChatGPT-User,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/chatgpt-user.json,"OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",2026-07-30
openai,OpenAI,GPTBot,User-agent: GPTBot,ai-training,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",GPTBot,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/gptbot.json,OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.,2026-07-30
openai,OpenAI,OAI-AdsBot,User-agent: OAI-AdsBot,advertising,true,not-published,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot",OAI-AdsBot,operator-documentation,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/adsbot.json,"OpenAI documents this token for checking pages submitted as ads and states the data collected is not used to train its models; its page does not say whether OAI-AdsBot follows robots.txt, so no behaviour is claimed here. No community pattern matches OpenAI's published string, so this row's pattern was authored from that page.",2026-07-30
openai,OpenAI,OAI-SearchBot,User-agent: OAI-SearchBot,search,true,obeys,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot",OAI-SearchBot,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/searchbot.json,"OpenAI recommends allowing OAI-SearchBot so a site can appear in ChatGPT search answers, and says a robots.txt change can take about 24 hours to take effect.",2026-07-30
perplexity,Perplexity,Perplexity-User,User-agent: Perplexity-User,user-triggered,true,user-triggered-may-ignore,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",Perplexity-User,crawler-user-agents,https://docs.perplexity.ai/docs/resources/perplexity-crawlers,published-ip-ranges,https://www.perplexity.com/perplexity-user.json,"Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules.",2026-07-30
perplexity,Perplexity,PerplexityBot,User-agent: PerplexityBot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",PerplexityBot\/,crawler-user-agents,https://docs.perplexity.ai/docs/resources/perplexity-crawlers,published-ip-ranges,https://www.perplexity.com/perplexitybot.json,Perplexity states this bot surfaces and links sites in Perplexity search results and is not used to crawl content for AI foundation models.,2026-07-30

About this API

Coverage, behavior and common use cases

Coverage & behavior

Two things, kept strictly apart because their provenance differs.

1500 user-agent patterns — the 1498 regular expressions in the crawler-user-agents project (MIT, pinned to one commit and verified by SHA-256), plus 2 patterns this repo authored for tokens an operator documents but that list does not carry. /api/crawlers lists them; /api/crawlers/detect?userAgent=… tells you what a string you have matches. Matching is case-sensitive and byte for byte: the upstream patterns encode their own case alternatives ([cC]laude[bB]ot), and 13 of them are ^-anchored, so googlebot/2.1 does not match and curl/8.7.1 does not match either. Both cases produce an explanatory warning rather than a silent no.

24 robots.txt tokens across 8 operators — /api/crawlers/robots-tokens is the curated half: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta and Amazon, every row read off that operator's own documentation page (last checked 2026-07-30) and carrying that page's URL. It is a curated subset per operator, not a complete catalogue — Google alone documents eleven crawlers on its common-crawlers page and we curate four; each row's docsUrl is the full list.

What is real, and what is deliberately null

operator, robotsToken, purpose, verification and note exist only on the 24 rows a first-party operator page states them for — with one named exception: on CCBot and meta-webindexer the purpose value is our classification rather than the operator's wording, and each of those rows says so in its own note, quoting what the operator actually wrote. On the other 1476 rows all five fields are null. We never derive a robots.txt token from a regular expression: ClaudeBot from [cC]laude[bB]ot would be a guess about capitalisation, and (^| )sentry\/ yields nothing at all. A wrong token in your robots.txt is worse than no token.
Operators without first-party crawler documentation (ByteDance's Bytespider, Mistral) get no curated row, and this page makes no claim about how they behave — we did not measure it and they did not publish it.
exampleUserAgent is non-null on 118 of 1500 rows, by a published rule: the rows a curated token points at, every row the community list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. That is not missing data — the field exists so you can paste a known-good string into /detect, which needs a well-chosen 118, not 1498. For anything else, paste a line from your own access log.
Google-Extended and Applebot-Extended are control tokens only (sendsRequests: false). Nothing ever sends them, which is also why neither carries an example string — the community list does carry one for Google-Extended, and we drop it rather than publish a string no request sends.
tags is the community project's classification, not the vendors' self-descriptions, and addedOn is when a pattern entered that list, not when the crawler started operating.
robotsTxtBehavior is the operator's own published claim about itself. We do not test compliance, publish no accusation, and record not-published where a page simply does not say.

It is not a security control

A User-Agent header is caller-supplied text. isKnownCrawler: false means "no pattern in this snapshot matched" — never "a human sent this", because a crawler impersonating a browser is exactly what a pattern list cannot see. The real check is the operator's published IP ranges or reverse DNS, and every curated row hands you that URL in verification. We never fetch it, and there is no request-time network here at all.

Nearby endpoints

/api/robots-txt/check is the other half of the workflow: list the tokens here, then paste a robots.txt and test the rule there. /api/reverse-dns builds the in-addr.arpa / ip6.arpa name you need for the PTR check. And do not confuse this with /api/user-agents, which generates plausible fake browser strings for fixtures — this endpoint carries a pinned list of real crawler patterns and never invents one.

Use it for

Decide which AI-training crawlers to disallow, then generate the robots.txt lines to paste
Classify user agents in access logs or edge middleware without hand-maintaining a bot regex list
Look up which company operates a bot you saw, and get its own IP-range list to verify the request
Explain a bot-detection false negative: a case-only or whitespace-only mismatch is reported, not swallowed
Audit an existing robots.txt against the tokens the eight biggest operators actually publish

Frequently asked questions

Practical answers about this endpoint 5

Which user agents should I block to stop AI training?

The tokens to disallow to stop AI training are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent — /api/crawlers/robots-tokens?purpose=ai-training returns them with the exact User-agent: line for each. CCBot is our classification rather than the operator's wording — the other 5 labels come from the operator's own page — because Common Crawl's page describes only "an open repository of web crawl data" and never mentions training — its corpus is simply the canonical public training corpus, and the row's note says exactly that. Amazon's Amazonbot is deliberately general rather than ai-training: Amazon states it "is used to improve our products and services" and names training only as a possible further use ("may be used to train Amazon AI models"), so disallow it too if you want that use excluded.

Does a matching user agent prove the request came from that crawler?

No. A User-Agent header is caller-supplied text, so anyone can send any string. The real check is the operator's published IP ranges or reverse DNS, and every curated row hands you that URL in verification — this API never fetches it. Use /api/reverse-dns to build the PTR name you need.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

Training, search indexing and user-triggered fetching, all three documented by OpenAI: GPTBot crawls content that may train its models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches a page because a person asked. OpenAI states robots.txt rules may not apply to that last one. Anthropic and Perplexity publish the same kind of split.

Is Google-Extended a crawler?

No — it is a control token only. Google's own page states it "doesn't have a separate HTTP request user agent string", that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect Search inclusion or ranking. Applebot-Extended works the same way, so both carry sendsRequests: false and no example string.

Does this endpoint check my site or fetch my robots.txt?

No. Nothing is fetched while your request is served. You supply the user-agent string to /api/crawlers/detect, and /api/robots-txt/check tests a robots.txt file you paste — neither route touches the network.

Standards & references

Primary sources behind the contract

Useful next tools in the catalog

robots.txt Tester and Parser API REAL DATA

Test a URL path against a robots.txt you paste, for any crawler, and see the winning rule under RFC 9309 most-octets precedence.

Random User Agent API MOCK DATA

Realistic, coherent browser user agents — filter by browser, OS and device; impossible combos like Safari-on-Windows get a clear 400, never a fake string.

Reverse DNS PTR Name Converter API REAL DATA

Encode IPv4 and IPv6 addresses as canonical reverse-DNS PTR owner names, or decode complete in-addr.arpa and ip6.arpa names without a DNS query.

Random Internet Data API MOCK DATA

Random internet building blocks — valid IPv4/IPv6 addresses, locally-administered MACs, non-resolvable .test domains and URLs, slugs, ports and TLDs.

HTTP Field Name Registry API REAL DATA

Complete pinned IANA HTTP field-name registry with status, Structured Field type, references, search and case-insensitive lookup.

GET /api/crawlers List known crawler and AI-bot user-agent patterns Returns a list

Request parameters

Response

Route reference

Documented examples

GET /api/crawlers/detect Check a user-agent string against every known crawler pattern Computed answer

Request parameters

Response

Route reference

Documented examples

GET /api/crawlers/robots-tokens The robots.txt tokens each operator publishes for its own crawlers Returns a list

Request parameters

Response

Route reference

Documented examples

About this API

Coverage & behavior

What is real, and what is deliberately null

It is not a security control

Nearby endpoints

Use it for

Frequently asked questions

Standards & references

Related APIs