Known crawler and AI-bot user agents: the robots.txt token each operator publishes, and what a user-agent string you supply actually matches.
Authoritative reference data or standards computation
All 1502 bundled patterns, ordered by 'pattern' ascending on UTF-16 code units so paging with 'offset' is stable. Filters compose with AND; a combination that matches nothing returns an empty list plus a warning, never a substitute row.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
tag | enum | default: any allowed: search-engine | advertising | feed-reader | http-library | social-preview | archiver | seo | monitoring | scanner | ai-crawler | academic | browser-automation | any example: ai-crawler | Keep only rows the upstream project gives this tag. A row can carry several. The 4 operator-documentation rows carry no tags and are never returned by this filter. |
operator | enum | default: any allowed: openai | anthropic | perplexity | google | apple | common-crawl | meta | amazon | mistral | any example: anthropic | Keep only rows curated to this operator — the ones whose own documentation states the token. |
documented | boolean | example: true | true keeps only rows whose operator documents the bot itself (operator is not null); false keeps only the uncurated community rows. Combining false with an operator is a 400, because nothing can satisfy both. |
search | string | allowed: ≤ 64 example: perplexity | Case-insensitive substring match against 'pattern', 'robotsToken' and 'operator'. At least 2 characters — a single character matches almost the whole list and is rejected with a 400. |
offset | int | default: 0 allowed: 0 – 5000 example: 100 | Skip this many matches before 'count' is applied — the list is far longer than count's ceiling of 100, so ?count=100&offset=100 returns the second hundred. |
tag enum Keep only rows the upstream project gives this tag. A row can carry several. The 4 operator-documentation rows carry no tags and are never returned by this filter.
operator enum Keep only rows curated to this operator — the ones whose own documentation states the token.
documented boolean true keeps only rows whose operator documents the bot itself (operator is not null); false keeps only the uncurated community rows. Combining false with an operator is a 400, because nothing can satisfy both.
search string Case-insensitive substring match against 'pattern', 'robotsToken' and 'operator'. At least 2 characters — a single character matches almost the whole list and is rejected with a 400.
offset int Skip this many matches before 'count' is applied — the list is far longer than count's ceiling of 100, so ?count=100&offset=100 returns the second hundred.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
count | int | default: 50 allowed: 1 – 100 example: 3 | Limit the number of returned records (1–100). Defaults to 50. |
fields | list | example: pattern,patternSource | Return only these fields (comma-separated). Mutually exclusive with 'exclude'. |
exclude | list | example: addedOn | Return all fields except these (comma-separated). |
format | enum | default: json allowed: json | ndjson | csv example: csv | Response format: json envelope, ndjson (one record per line) or csv. |
pretty | boolean | default: false example: true | Pretty-print the JSON response. |
unwrap | boolean | default: false example: true | Drop the envelope: return the raw array/object without data/meta wrapper. |
count int Limit the number of returned records (1–100). Defaults to 50.
fields list Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
exclude list Return all fields except these (comma-separated).
format enum Response format: json envelope, ndjson (one record per line) or csv.
pretty boolean Pretty-print the JSON response.
unwrap boolean Drop the envelope: return the raw array/object without data/meta wrapper.
| Field | Type | Description | Example |
|---|---|---|---|
pattern | string | The regular expression this row is matched by, verbatim. Matched case-sensitively with RegExp.test, because the upstream patterns encode their own case alternatives ([cC]laude[bB]ot). 13 patterns are anchored to the start of the string, so leading whitespace changes the answer. | [cC]laude[bB]ot |
patternSource | string | crawler-user-agents (the pinned community list, 1,498 rows) or operator-documentation (a token the operator documents that the community list does not carry, so the exact literal pattern was authored from the operator's own page — exactly 4 rows: OAI-AdsBot, Applebot-Extended, MistralAI-Index and MistralAI-Training). | crawler-user-agents |
tags | string[] | The upstream project's own classification, from its fixed 12-value vocabulary, in canonical order. It is the maintainers' judgement, not the operator's self-description. Empty on the 4 operator-documentation rows — the project has not classified them and we do not classify on its behalf, so those rows appear under no tag filter. | ["ai-crawler"] |
operator nullable | string | Company or project that operates this crawler — set only for rows curated against the operator's own published documentation. null on the 1,475 community rows: we do not guess who runs a bot. | Anthropic |
operatorId nullable | string | The value to pass back as ?operator=. null whenever operator is. | anthropic |
robotsToken nullable | string | The exact token to write after 'User-agent:', as the operator publishes it. null on every uncurated row — a token derived from a regular expression would be a guess about capitalisation, and a wrong token in your robots.txt is worse than no token. | ClaudeBot |
purpose nullable | string | What this crawler is for: search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. A randomapi.dev classification based on the operator's published purpose, with the source wording preserved in the note. On 3 non-literal mappings (Amazonbot, CCBot and meta-webindexer), the note explicitly labels the category as a randomapi.dev classification. null when uncurated. | ai-training |
exampleUserAgent nullable | string | One real User-Agent string, observed in the wild, that this pattern matches — the longest entry the upstream snapshot carries for this row. Paste it into /api/crawlers/detect. Non-null on 118 of the 1,502 rows by a published rule: the rows a curated operator token points at, every row the upstream list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. For any other pattern, paste a line from your own access log instead. Never the operator's own published template — that is fullUserAgentExample on /robots-tokens. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
referenceUrl nullable | string | The reference link the community list carries for this crawler, copied and NOT verified: 162 rows have none, 298 are plain http://, 2 are malformed upstream (a missing or truncated scheme) and are still copied byte for byte because inventing a scheme would be inventing provenance, and for a few hundred it points at a third-party bot directory, a forge or a forum thread rather than the operator's own site. Use operatorDocsUrl when you need a citable page. | http://www.google.com/bot.html |
operatorDocsUrl nullable | string | The operator's own crawler-documentation page — the primary citable source for curated rows. Supplemental first-party policy pages used by behavioural notes are cited in this endpoint's data sources. | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
verification nullable | object | How to check a request really came from this operator: { method, url }, where method is published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published, and url is the operator's own list (or null). We never fetch it — it is a URL for you to use. Present on curated rows only. | {"method":"published-ip-ranges","url":"https://claude.com/crawling/bots.json"} |
note nullable | string | One curated caveat in our own words, where the operator's documentation says something that changes how you use the token. Curated rows only. | Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity. |
addedOn nullable | string (YYYY-MM-DD) | Date the pattern entered the community list — NOT the date the crawler began operating. null on 57 rows. | 2015-04-14 |
pattern string The regular expression this row is matched by, verbatim. Matched case-sensitively with RegExp.test, because the upstream patterns encode their own case alternatives ([cC]laude[bB]ot). 13 patterns are anchored to the start of the string, so leading whitespace changes the answer.
example: [cC]laude[bB]ot
patternSource string crawler-user-agents (the pinned community list, 1,498 rows) or operator-documentation (a token the operator documents that the community list does not carry, so the exact literal pattern was authored from the operator's own page — exactly 4 rows: OAI-AdsBot, Applebot-Extended, MistralAI-Index and MistralAI-Training).
example: crawler-user-agents
tags string[] The upstream project's own classification, from its fixed 12-value vocabulary, in canonical order. It is the maintainers' judgement, not the operator's self-description. Empty on the 4 operator-documentation rows — the project has not classified them and we do not classify on its behalf, so those rows appear under no tag filter.
example: ["ai-crawler"]
operator string nullable Company or project that operates this crawler — set only for rows curated against the operator's own published documentation. null on the 1,475 community rows: we do not guess who runs a bot.
example: Anthropic
operatorId string nullable The value to pass back as ?operator=. null whenever operator is.
example: anthropic
robotsToken string nullable The exact token to write after 'User-agent:', as the operator publishes it. null on every uncurated row — a token derived from a regular expression would be a guess about capitalisation, and a wrong token in your robots.txt is worse than no token.
example: ClaudeBot
purpose string nullable What this crawler is for: search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. A randomapi.dev classification based on the operator's published purpose, with the source wording preserved in the note. On 3 non-literal mappings (Amazonbot, CCBot and meta-webindexer), the note explicitly labels the category as a randomapi.dev classification. null when uncurated.
example: ai-training
exampleUserAgent string nullable One real User-Agent string, observed in the wild, that this pattern matches — the longest entry the upstream snapshot carries for this row. Paste it into /api/crawlers/detect. Non-null on 118 of the 1,502 rows by a published rule: the rows a curated operator token points at, every row the upstream list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. For any other pattern, paste a line from your own access log instead. Never the operator's own published template — that is fullUserAgentExample on /robots-tokens.
example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
referenceUrl string nullable The reference link the community list carries for this crawler, copied and NOT verified: 162 rows have none, 298 are plain http://, 2 are malformed upstream (a missing or truncated scheme) and are still copied byte for byte because inventing a scheme would be inventing provenance, and for a few hundred it points at a third-party bot directory, a forge or a forum thread rather than the operator's own site. Use operatorDocsUrl when you need a citable page.
example: http://www.google.com/bot.html
operatorDocsUrl string nullable The operator's own crawler-documentation page — the primary citable source for curated rows. Supplemental first-party policy pages used by behavioural notes are cited in this endpoint's data sources.
example: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
verification object nullable How to check a request really came from this operator: { method, url }, where method is published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published, and url is the operator's own list (or null). We never fetch it — it is a URL for you to use. Present on curated rows only.
example: {"method":"published-ip-ranges","url":"https://claude.com/crawling/bots.json"}
note string nullable One curated caveat in our own words, where the operator's documentation says something that changes how you use the token. Curated rows only.
example: Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity.
addedOn string (YYYY-MM-DD) nullable Date the pattern entered the community list — NOT the date the crawler began operating. null on 57 rows.
example: 2015-04-14
/api/crawlers?tag=ai-crawler&count=100 {
"data": [
{
"pattern": "AI2Bot",
"patternSource": "crawler-user-agents",
"tags": [
"ai-crawler",
"academic"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": "AI2Bot",
"referenceUrl": "https://datadome.co/bots/ai2bot/",
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": "2026-04-17"
},
{
"pattern": "AISearchBot",
"patternSource": "crawler-user-agents",
"tags": [
"search-engine",
"ai-crawler"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": null,
"referenceUrl": null,
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": null
},
{
"pattern": "Amzn-SearchBot",
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers?documented=true {
"data": [
{
"pattern": "Amazonbot",
"patternSource": "crawler-user-agents",
"tags": [
"search-engine"
],
"operator": "Amazon",
"operatorId": "amazon",
"robotsToken": "Amazonbot",
"purpose": "general",
"exampleUserAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)",
"referenceUrl": "https://developer.amazon.com/support/amazonbot",
"operatorDocsUrl": "https://developer.amazon.com/amazonbot",
"verification": {
"method": "published-ip-ranges",
"url": "https://developer.amazon.com/amazonbot/ip-addresses/"
},
"note": "Amazon states this crawler \"is used to improve our products and services\" and that the data \"may be used to train Amazon AI models\". Purpose 'general' is the randomapi.dev classification of that broad wording; the possible training use stays visible here rather than being promoted into the crawler's sole purpose. Amazon also states its crawlers do not support the crawl-delay directive. Chrome/W.X.Y.Z is Amazon's own placeholder.",
"addedOn": "2020-03-02"
},
{
"pattern": "Amzn-SearchBot",
"patternSource": "crawler-user-agents",
"tags": [
"ai-crawler"
],
"operator": "Amazon",
"operatorId": "amazon",
"robotsToken": "Amzn-SearchBot",
"purpose": "search",
"exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36",
"referenceUrl": "https://developer.amazon.com/amazonbot",
"operatorDocsUrl": "https://developer.amazon.com/amazonbot",
"verification": {
"method": "published-ip-ranges",
"url": "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/"
},
"note": "Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers?operator=perplexity {
"data": [
{
"pattern": "Perplexity-User",
"patternSource": "crawler-user-agents",
"tags": [
"search-engine",
"ai-crawler"
],
"operator": "Perplexity",
"operatorId": "perplexity",
"robotsToken": "Perplexity-User",
"purpose": "user-triggered",
"exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",
"referenceUrl": "https://docs.perplexity.ai/guides/bots",
"operatorDocsUrl": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
"verification": {
"method": "published-ip-ranges",
"url": "https://www.perplexity.com/perplexity-user.json"
},
"note": "Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules.",
"addedOn": "2026-04-07"
},
{
"pattern": "PerplexityBot\\/",
"patternSource": "crawler-user-agents",
"tags": [
"search-engine",
"ai-crawler"
],
"operator": "Perplexity",
"operatorId": "perplexity",
"robotsToken": "PerplexityBot",
"purpose": "search",
"exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
"referenceUrl": "https://docs.perplexity.ai/docs/perplexitybot",
"operatorDocsUrl": "https://docs.perplexity.ai/docs/resources/perplexity-crawlers",
"verification": {
"method": "published-ip-ranges",
"url": "https://www.perplexity.com/perplexitybot.json"
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers?tag=seo&count=50&offset=50 {
"data": [
{
"pattern": "Caliperbot",
"patternSource": "crawler-user-agents",
"tags": [
"seo"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": null,
"referenceUrl": "http://www.conductor.com/caliperbot",
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": "2019-03-02"
},
{
"pattern": "CapitalOneShopping",
"patternSource": "crawler-user-agents",
"tags": [
"seo"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": null,
"referenceUrl": "https://datadome.co/bots/capital-one-shopping-bot/",
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": "2026-04-17"
},
{
"pattern": "CentComBot\\/",
"patternSource": "crawler-user-agents",
"tags": [
…
]
}
} /api/crawlers?tag=ai-crawler&count=100&format=csv pattern,patternSource,tags,operator,operatorId,robotsToken,purpose,exampleUserAgent,referenceUrl,operatorDocsUrl,verification,note,addedOn,verification.method,verification.url
AI2Bot,crawler-user-agents,"[""ai-crawler"",""academic""]",,,,,AI2Bot,https://datadome.co/bots/ai2bot/,,,,2026-04-17,,
AISearchBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,,,,,,,,
Amzn-SearchBot,crawler-user-agents,"[""ai-crawler""]",Amazon,amazon,Amzn-SearchBot,search,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36",https://developer.amazon.com/amazonbot,https://developer.amazon.com/amazonbot,,"Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",2026-04-26,published-ip-ranges,https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
Amzn-User,crawler-user-agents,"[""ai-crawler""]",Amazon,amazon,Amzn-User,user-triggered,Mozilla/5.0 (compatible; Amzn-User; +https://developer.amazon.com/),https://developer.amazon.com/amazonbot,https://developer.amazon.com/amazonbot,,"Amazon states this agent fetches live web information on a user's behalf and, because its actions can be user-initiated, may not follow all robots.txt directives. That is narrower than a blanket claim that it ignores robots.txt. Amazon also states Amzn-User does not crawl content for generative AI model training.",2026-04-26,published-ip-ranges,https://developer.amazon.com/amazonbot/live-ip-addresses/
Anomura,crawler-user-agents,"[""ai-crawler""]",,,,,Anomura/1.2 (+https://www.direqt.ai),https://docs.direqt-search.com/direqt-bots/direqt-crawlers-and-user-agents,,,,2026-04-26,,
ApifyBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ApifyBot/1.0),https://knownagents.com/agents/apifybot,,,,2026-04-26,,
ApifyWebsiteContentCrawler,crawler-user-agents,"[""ai-crawler""]",,,,,ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler),https://apify.com/apify/website-content-crawler,,,,2026-04-26,,
Aranet-SearchBot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36",https://aranet.ai/bot,,,,2026-04-26,,
AzureAI-SearchBot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;",https://azure.microsoft.com/en-us/products/ai-services,,,,2026-04-26,,
Brightbot,crawler-user-agents,"[""ai-crawler""]",,,,,Brightbot 1.0,https://brightdata.com/brightbot,,,,2026-04-26,,
Bytespider,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com",https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent,,,,2019-11-11,,
CCBot,crawler-user-agents,"[""ai-crawler""]",Common Crawl,common-crawl,CCBot,ai-training,CCBot/2.0 (https://commoncrawl.org/faq/),http://www.commoncrawl.org/bot.html,https://commoncrawl.org/ccbot,,"Common Crawl's page describes CCBot only as producing ""an open repository of web crawl data that is universally accessible and analyzable by anyone"" — it does not mention AI training or characterize the repository as a training corpus. Purpose 'ai-training' is therefore a randomapi.dev classification for a crawler-use-preference list, not Common Crawl's wording. The page also warns that other crawlers falsely identify themselves as CCBot, and that real CCBot traffic reverse-resolves under crawl.commoncrawl.org (IPv4 only).",2012-02-05,reverse-dns-and-ip-ranges,https://index.commoncrawl.org/ccbot.json
Channel3Bot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Channel3Bot/1.0; +https://trychannel3.com/channel3bot),https://trychannel3.com/channel3bot,,,,2026-04-26,,
ChatGLM-Spider,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/),https://chatglm.cn/,,,,2026-04-17,,
ChatGPT-User,crawler-user-agents,"[""ai-crawler""]",OpenAI,openai,ChatGPT-User,user-triggered,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",https://openai.com/bot,https://developers.openai.com/api/docs/bots,,"OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",2024-04-19,published-ip-ranges,https://openai.com/chatgpt-user.json
Claude-SearchBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",Anthropic,anthropic,Claude-SearchBot,search,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)",https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,,"Anthropic states this bot analyses content to improve search result quality, and that disabling it prevents indexing for search and may reduce a site's visibility in search results.",2026-04-07,published-ip-ranges,https://claude.com/crawling/bots.json
Claude-User,crawler-user-agents,"[""ai-crawler""]",Anthropic,anthropic,Claude-User,user-triggered,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)",https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,,"Anthropic states this agent fetches pages when a person asks Claude a question, and lists it among the bots that honour robots.txt — unlike several other operators' user-triggered fetchers.",2026-04-07,published-ip-ranges,https://claude.com/crawling/bots.json
Claude-Web,crawler-user-agents,"[""ai-crawler""]",,,,,Claude-Web/1.0 (web crawler; +https://www.anthropic.com/; bots@anthropic.com),https://anthropic.com,,,,2026-04-07,,
Cloudflare-AutoRAG,crawler-user-agents,"[""ai-crawler""]",,,,,Cloudflare-AutoRAG (https://developers.cloudflare.com/autorag; autorag@cloudflare.com),https://developers.cloudflare.com/autorag,,,,2026-04-26,,
DeepSeekBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot),http://deepseek.com/bot,,,,2026-04-26,,
Devin,crawler-user-agents,"[""ai-crawler"",""browser-automation""]",,,,,"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36; Devin/1.0; +https://devin.ai",https://docs.devin.ai/get-started/devin-intro,,,,2026-04-17,,
DuckAssistBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html),https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot,,,,2026-04-07,,
ExteContextCrawl,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; ExteContextCrawl/1.0; +http://crawl001.exte.ai),http://crawl001.exte.ai/,,,,2026-04-26,,
FacebookBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/facebookbot/),https://developers.facebook.com/docs/sharing/bot/,,,,2026-04-26,,
FirecrawlAgent,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; FirecrawlAgent; +https://firecrawl.dev/),https://firecrawl.dev/,,,,2026-04-26,,
Flyriverbot,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Flyriverbot/1.1 (+https://www.flyriver.com/; AI Content Source Check)",https://flyriver.com/crawler,,,,2026-04-26,,
GPTBot,crawler-user-agents,"[""ai-crawler""]",OpenAI,openai,GPTBot,ai-training,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)",https://platform.openai.com/docs/gptbot,https://developers.openai.com/api/docs/bots,,OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.,2023-08-09,published-ip-ranges,https://openai.com/gptbot.json
Gemini-Deep-Research,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36",https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers,,,,2026-04-07,,
Google-CloudVertexBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",Google,google,Google-CloudVertexBot,site-owner-requested,,https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-cloudvertexbot,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,,"Google states preferences for this token affect crawls that site owners themselves request for building Vertex AI Agents, and have no effect on Google Search. Google publishes only the user-agent substring for it, not a full string, so fullUserAgentExample is null.",2026-04-17,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Google-Extended,crawler-user-agents,"[""ai-crawler""]",Google,google,Google-Extended,ai-training,,https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,,"Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect a site's inclusion in Google Search nor act as a ranking signal. Google also states the token governs two distinct things, not just one: whether crawled content may be used for training future generations of the Gemini models behind Gemini Apps and the Vertex AI API, AND grounding — supplying content from the Google Search index to the model at prompt time — in Gemini Apps and Grounding with Google Search on Vertex AI. A disallow preference applies to both named uses; it is not a separate crawler or a retroactive deletion mechanism. Nothing ever sends this string, so the verification link is for Google's actual crawler traffic.",2026-04-07,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Google-NotebookLM,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Google-NotebookLM; +https://notebooklm.google.com/),https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers,,,,2026-04-26,,
HenkBot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; HenkBot/1.0; +https://valyu.ai/crawler),https://valyu.ai/crawler,,,,2026-04-26,,
ImageMind,crawler-user-agents,"[""ai-crawler""]",,,,,ImageMind,https://datadome.co/bots/imagemind/,,,,2026-04-17,,
Kangaroo Bot,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; Kangaroo Bot/1.0; +http://www.kangaroo.com),https://kangaroollm.com.au/kangaroo-bot/,,,,2026-04-26,,
KendraBot,crawler-user-agents,"[""search-engine"",""ai-crawler""]",,,,,KendraBot,https://docs.aws.amazon.com/kendra/latest/dg/what-is-kendra.html,,,,2026-04-17,,
KunatoCrawler,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (compatible; KunatoCrawler/1.0; +http://kunato.ai/bot.html),http://kunato.ai/bot.html,,,,2026-04-26,,
LinerBot\/,crawler-user-agents,"[""ai-crawler""]",,,,,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinerBot/1.0; +https://docs.getliner.com/docs/linerbot)",https://docs.getliner.com/docs/linerbot,,,,2026-04-17,,
LinkupBot,crawler-user-agents,"[""ai-crawler""]",,,,,LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so),https://linkup.so/bot,,,,2026-04-26,,
MistralAI-User,crawler-user-agents,"[""ai-crawler""]",Mistral,mistral,MistralAI-User,user-triggered,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)",,https://docs.mistral.ai/robots,,"Mistral states this agent supports user actions in Vibe, governs which sites those user-requested visits can access, and is not used for automatic web crawling or generative AI training.",2026-04-07,published-ip-ranges,https://mistral.ai/mistralai-user-ips.json
…
newsai\/,crawler-user-agents,"[""feed-reader"",""ai-crawler""]",,,,,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36",https://knownagents.com/agents/newsai,,,,2026-04-14,,
semantic-visions,crawler-user-agents,"[""ai-crawler""]",,,,,Mozilla/5.0 (Linux; CentOS; compatible; semantic-visions-discovery; HTTPClient 4.5),https://semantic-visions.com/,,,,2026-04-26,,
Scans the string you send against all 1502 bundled patterns and returns every match. Pure function of 'userAgent': no clock, no network, no state. The string is taken byte for byte — nothing is trimmed, because 13 patterns are anchored and a stray leading space genuinely changes the verdict.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
userAgent
required
| string | example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | The User-Agent header value to test, exactly as received. Taken byte for byte: leading and trailing whitespace is kept (and warned about) rather than stripped, matching is case-sensitive, and a CR or LF is a 400 because an HTTP field value is a single line. Capped at 1024 characters, which every string bundled here fits inside. |
userAgent string required The User-Agent header value to test, exactly as received. Taken byte for byte: leading and trailing whitespace is kept (and warned about) rather than stripped, matching is case-sensitive, and a CR or LF is a 400 because an HTTP field value is a single line. Capped at 1024 characters, which every string bundled here fits inside.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
fields | list | example: userAgent,isKnownCrawler | Return only these fields (comma-separated). Mutually exclusive with 'exclude'. |
exclude | list | example: robotsTokens | Return all fields except these (comma-separated). |
pretty | boolean | default: false example: true | Pretty-print the JSON response. |
unwrap | boolean | default: false example: true | Drop the envelope: return the raw array/object without data/meta wrapper. |
fields list Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
exclude list Return all fields except these (comma-separated).
pretty boolean Pretty-print the JSON response.
unwrap boolean Drop the envelope: return the raw array/object without data/meta wrapper.
| Field | Type | Description | Example |
|---|---|---|---|
userAgent | string | The string that was tested, echoed exactly as supplied — nothing is trimmed. | CCBot/2.0 (https://commoncrawl.org/faq/) |
isKnownCrawler | boolean | Whether at least one bundled pattern matched. false means 'no pattern in this snapshot matched' — it never means the request came from a human. Anyone can send any User-Agent, and crawlers that disguise themselves as browsers are exactly the ones a string list cannot see. | true |
matchCount | integer | Exact number of patterns that matched, even when matches is capped at 25. | 1 |
matches | object[] | Every matching row in the same 13-field shape as /api/crawlers, ordered curated-first, then longest pattern first, then pattern ascending. Capped at 25 entries with a warning; matchCount stays exact. Empty array when nothing matched. | [{"pattern":"CCBot","robotsToken":"CCBot","operator":"Common Crawl"}] |
operators | string[] | Distinct curated operators among the returned matches, in first-appearance order. Empty when no matched row is curated. | ["Common Crawl"] |
robotsTokens | string[] | Distinct curated robotsToken values among the returned matches — the lines to write in robots.txt if you want to control this traffic. Empty when nothing curated matched. | ["CCBot"] |
userAgent string The string that was tested, echoed exactly as supplied — nothing is trimmed.
example: CCBot/2.0 (https://commoncrawl.org/faq/)
isKnownCrawler boolean Whether at least one bundled pattern matched. false means 'no pattern in this snapshot matched' — it never means the request came from a human. Anyone can send any User-Agent, and crawlers that disguise themselves as browsers are exactly the ones a string list cannot see.
example: true
matchCount integer Exact number of patterns that matched, even when matches is capped at 25.
example: 1
matches object[] Every matching row in the same 13-field shape as /api/crawlers, ordered curated-first, then longest pattern first, then pattern ascending. Capped at 25 entries with a warning; matchCount stays exact. Empty array when nothing matched.
example: [{"pattern":"CCBot","robotsToken":"CCBot","operator":"Common Crawl"}]
operators string[] Distinct curated operators among the returned matches, in first-appearance order. Empty when no matched row is curated.
example: ["Common Crawl"]
robotsTokens string[] Distinct curated robotsToken values among the returned matches — the lines to write in robots.txt if you want to control this traffic. Empty when nothing curated matched.
example: ["CCBot"]
/api/crawlers/detect?userAgent=Mozilla%2F5.0%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%3B%20compatible%3B%20GPTBot%2F1.4%3B%20%2Bhttps%3A%2F%2Fopenai.com%2Fgptbot {
"data": {
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
"isKnownCrawler": true,
"matchCount": 1,
"matches": [
{
"pattern": "GPTBot",
"patternSource": "crawler-user-agents",
"tags": [
"ai-crawler"
],
"operator": "OpenAI",
"operatorId": "openai",
"robotsToken": "GPTBot",
"purpose": "ai-training",
"exampleUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)",
"referenceUrl": "https://platform.openai.com/docs/gptbot",
"operatorDocsUrl": "https://developers.openai.com/api/docs/bots",
"verification": {
"method": "published-ip-ranges",
"url": "https://openai.com/gptbot.json"
},
"note": "OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.",
"addedOn": "2023-08-09"
}
],
"operators": [
"OpenAI"
],
"robotsTokens": [
"GPTBot"
]
},
"meta": {
"endpoint": "crawlers",
"route": "/detect",
"params": {
"userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
},
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/detect?userAgent=Mozilla%2F5.0%20(Windows%20NT%2010.0%3B%20Win64%3B%20x64)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F138.0.0.0%20Safari%2F537.36 {
"data": {
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
"isKnownCrawler": false,
"matchCount": 0,
"matches": [],
"operators": [],
"robotsTokens": []
},
"meta": {
"endpoint": "crawlers",
"route": "/detect",
"params": {
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
},
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/detect?userAgent=linkdexbot%2FNutch-1.0-dev%20(http%3A%2F%2Fwww.linkdex.com%2F%3B%20crawl%20at%20linkdex%20dot%20com) {
"data": {
"userAgent": "linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)",
"isKnownCrawler": true,
"matchCount": 2,
"matches": [
{
"pattern": "linkdex",
"patternSource": "crawler-user-agents",
"tags": [
"seo"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": null,
"referenceUrl": "http://www.linkdex.com",
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": "2011-01-06"
},
{
"pattern": "Nutch",
"patternSource": "crawler-user-agents",
"tags": [
"search-engine"
],
"operator": null,
"operatorId": null,
"robotsToken": null,
"purpose": null,
"exampleUserAgent": null,
"referenceUrl": null,
"operatorDocsUrl": null,
"verification": null,
"note": null,
"addedOn": null
}
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/detect?userAgent=googlebot%2F2.1 {
"data": {
"userAgent": "googlebot/2.1",
"isKnownCrawler": false,
"matchCount": 0,
"matches": [],
"operators": [],
"robotsTokens": []
},
"meta": {
"endpoint": "crawlers",
"route": "/detect",
"params": {
"userAgent": "googlebot/2.1"
},
"generatedAt": "2026-09-15T18:48:07.000Z",
"warnings": [
"No pattern matched, but 1 would match if case were ignored: 'Googlebot\\/'. The bundled patterns are matched case-sensitively because they encode their own case alternatives (for example '[cC]laude[bB]ot'), so a lowercased log line does not match — check the capitalisation of the string you sent."
]
}
}The curated half of this endpoint: 27 tokens from 9 operators, each read off that operator's own documentation page and last checked on 2026-08-27. A curated subset per operator, not a complete catalogue — follow each row's 'docsUrl' for the operator's full list. Ordered by operatorId then token.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
operator | enum | default: any allowed: openai | anthropic | perplexity | google | apple | common-crawl | meta | amazon | mistral | any example: openai | Only this operator's tokens. |
purpose | enum | default: any allowed: search | ai-training | user-triggered | advertising | site-owner-requested | link-preview | general | any example: ai-training | Only tokens recorded with this purpose. The value restates the operator's own published wording on every row except Amazonbot, CCBot and meta-webindexer, where the row's note says the label is ours. |
robotsTxtBehavior | enum | default: any allowed: obeys | obeys-except-integrity-checks | user-triggered-may-ignore | any example: user-triggered-may-ignore | Only tokens whose operator publishes this robots.txt behaviour about itself. This is policy metadata, not independent verification of crawler compliance. |
operator enum Only this operator's tokens.
purpose enum Only tokens recorded with this purpose. The value restates the operator's own published wording on every row except Amazonbot, CCBot and meta-webindexer, where the row's note says the label is ours.
robotsTxtBehavior enum Only tokens whose operator publishes this robots.txt behaviour about itself. This is policy metadata, not independent verification of crawler compliance.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
count | int | default: all matches allowed: 1 – 100 example: 3 | Limit the number of returned records (1–100). Defaults to all matches. |
fields | list | example: operatorId,operator | Return only these fields (comma-separated). Mutually exclusive with 'exclude'. |
exclude | list | example: verifiedOn | Return all fields except these (comma-separated). |
format | enum | default: json allowed: json | ndjson | csv example: csv | Response format: json envelope, ndjson (one record per line) or csv. |
pretty | boolean | default: false example: true | Pretty-print the JSON response. |
unwrap | boolean | default: false example: true | Drop the envelope: return the raw array/object without data/meta wrapper. |
count int Limit the number of returned records (1–100). Defaults to all matches.
fields list Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
exclude list Return all fields except these (comma-separated).
format enum Response format: json envelope, ndjson (one record per line) or csv.
pretty boolean Pretty-print the JSON response.
unwrap boolean Drop the envelope: return the raw array/object without data/meta wrapper.
| Field | Type | Description | Example |
|---|---|---|---|
operatorId | string | Filter value for ?operator=. | openai |
operator | string | Operator's name as it writes it. | OpenAI |
token | string | The token, in the operator's own spelling. RFC 9309, Section 2.2.1 requires crawlers to match product tokens case-insensitively, so the capitalisation is cosmetic; every token here also satisfies that section's product-token grammar (letters, '_' and '-' only), so no conforming parser truncates it. | GPTBot |
robotsTxtLine | string | The line to paste: 'User-agent: ' + token. Add your own Disallow: line under it, then test the whole file with /api/robots-txt/check. | User-agent: GPTBot |
purpose | string | search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. A randomapi.dev classification based on the operator's published purpose, with the source wording preserved in the note. On 3 non-literal mappings (Amazonbot, CCBot and meta-webindexer), the note explicitly labels the category as a randomapi.dev classification. | ai-training |
sendsRequests | boolean | Whether any request actually carries this string. false for Google-Extended and Applebot-Extended: control tokens only, so you will never see them in an access log. | true |
robotsTxtBehavior | string | obeys, obeys-except-integrity-checks or user-triggered-may-ignore, as the operator publishes it about itself. We do not measure compliance and this is not a claim that any crawler behaves as documented. | obeys |
fullUserAgentExample nullable | string | The User-Agent string the operator publishes for this token, verbatim — including the operator's own placeholder version tokens (Google and Amazon both publish Chrome/W.X.Y.Z). We never fill a placeholder in. null where the operator publishes none: Anthropic publishes tokens but no strings, Google publishes none for Google-Extended and only a substring for Google-CloudVertexBot. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
pattern | string | The bundled pattern that detects this token on /api/crawlers/detect, so the two routes are visibly the same universe. | GPTBot |
patternSource | string | crawler-user-agents when the community list already carries the exact pattern for this token; operator-documentation for the 4 missing exact patterns authored literally from first-party pages (OAI-AdsBot, Applebot-Extended, MistralAI-Index and MistralAI-Training). | crawler-user-agents |
docsUrl | string | The operator's own crawler-documentation page — the primary citable source for this row. Supplemental first-party policy pages used by behavioural notes are cited in this endpoint's data sources. | https://developers.openai.com/api/docs/bots |
verificationMethod | string | published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published. For the two control-only tokens this describes how to verify the operator's actual crawler traffic, since nothing ever sends the control token itself. | published-ip-ranges |
verificationUrl nullable | string | The operator's own IP-range list. Never fetched by this API. | https://openai.com/gptbot.json |
note | string | One caveat in our own words, where the operator's page says something that changes how the token behaves. | Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group. |
verifiedOn | string (YYYY-MM-DD) | The date this table was last checked against the operators' own pages. A module constant, identical on every row — never the current time. | 2026-08-27 |
operatorId string Filter value for ?operator=.
example: openai
operator string Operator's name as it writes it.
example: OpenAI
token string The token, in the operator's own spelling. RFC 9309, Section 2.2.1 requires crawlers to match product tokens case-insensitively, so the capitalisation is cosmetic; every token here also satisfies that section's product-token grammar (letters, '_' and '-' only), so no conforming parser truncates it.
example: GPTBot
robotsTxtLine string The line to paste: 'User-agent: ' + token. Add your own Disallow: line under it, then test the whole file with /api/robots-txt/check.
example: User-agent: GPTBot
purpose string search, ai-training, user-triggered, advertising, site-owner-requested, link-preview or general. A randomapi.dev classification based on the operator's published purpose, with the source wording preserved in the note. On 3 non-literal mappings (Amazonbot, CCBot and meta-webindexer), the note explicitly labels the category as a randomapi.dev classification.
example: ai-training
sendsRequests boolean Whether any request actually carries this string. false for Google-Extended and Applebot-Extended: control tokens only, so you will never see them in an access log.
example: true
robotsTxtBehavior string obeys, obeys-except-integrity-checks or user-triggered-may-ignore, as the operator publishes it about itself. We do not measure compliance and this is not a claim that any crawler behaves as documented.
example: obeys
fullUserAgentExample string nullable The User-Agent string the operator publishes for this token, verbatim — including the operator's own placeholder version tokens (Google and Amazon both publish Chrome/W.X.Y.Z). We never fill a placeholder in. null where the operator publishes none: Anthropic publishes tokens but no strings, Google publishes none for Google-Extended and only a substring for Google-CloudVertexBot.
example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
pattern string The bundled pattern that detects this token on /api/crawlers/detect, so the two routes are visibly the same universe.
example: GPTBot
patternSource string crawler-user-agents when the community list already carries the exact pattern for this token; operator-documentation for the 4 missing exact patterns authored literally from first-party pages (OAI-AdsBot, Applebot-Extended, MistralAI-Index and MistralAI-Training).
example: crawler-user-agents
docsUrl string The operator's own crawler-documentation page — the primary citable source for this row. Supplemental first-party policy pages used by behavioural notes are cited in this endpoint's data sources.
example: https://developers.openai.com/api/docs/bots
verificationMethod string published-ip-ranges, reverse-dns, reverse-dns-and-ip-ranges or none-published. For the two control-only tokens this describes how to verify the operator's actual crawler traffic, since nothing ever sends the control token itself.
example: published-ip-ranges
verificationUrl string nullable The operator's own IP-range list. Never fetched by this API.
example: https://openai.com/gptbot.json
note string One caveat in our own words, where the operator's page says something that changes how the token behaves.
example: Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group.
verifiedOn string (YYYY-MM-DD) The date this table was last checked against the operators' own pages. A module constant, identical on every row — never the current time.
example: 2026-08-27
/api/crawlers/robots-tokens?operator=openai {
"data": [
{
"operatorId": "openai",
"operator": "OpenAI",
"token": "ChatGPT-User",
"robotsTxtLine": "User-agent: ChatGPT-User",
"purpose": "user-triggered",
"sendsRequests": true,
"robotsTxtBehavior": "user-triggered-may-ignore",
"fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",
"pattern": "ChatGPT-User",
"patternSource": "crawler-user-agents",
"docsUrl": "https://developers.openai.com/api/docs/bots",
"verificationMethod": "published-ip-ranges",
"verificationUrl": "https://openai.com/chatgpt-user.json",
"note": "OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "openai",
"operator": "OpenAI",
"token": "GPTBot",
"robotsTxtLine": "User-agent: GPTBot",
"purpose": "ai-training",
"sendsRequests": true,
"robotsTxtBehavior": "obeys",
"fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
"pattern": "GPTBot",
"patternSource": "crawler-user-agents",
"docsUrl": "https://developers.openai.com/api/docs/bots",
"verificationMethod": "published-ip-ranges",
"verificationUrl": "https://openai.com/gptbot.json",
"note": "OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "openai",
"operator": "OpenAI",
"token": "OAI-AdsBot",
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/robots-tokens?purpose=ai-training {
"data": [
{
"operatorId": "anthropic",
"operator": "Anthropic",
"token": "ClaudeBot",
"robotsTxtLine": "User-agent: ClaudeBot",
"purpose": "ai-training",
"sendsRequests": true,
"robotsTxtBehavior": "obeys",
"fullUserAgentExample": null,
"pattern": "[cC]laude[bB]ot",
"patternSource": "crawler-user-agents",
"docsUrl": "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler",
"verificationMethod": "published-ip-ranges",
"verificationUrl": "https://claude.com/crawling/bots.json",
"note": "Anthropic states that restricting ClaudeBot signals that a site's future material should be excluded from its AI model training datasets, and that its bots honour robots.txt and the non-standard Crawl-delay extension. Anthropic publishes tokens but no user-agent strings.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "apple",
"operator": "Apple",
"token": "Applebot-Extended",
"robotsTxtLine": "User-agent: Applebot-Extended",
"purpose": "ai-training",
"sendsRequests": false,
"robotsTxtBehavior": "obeys",
"fullUserAgentExample": null,
"pattern": "Applebot-Extended",
"patternSource": "operator-documentation",
"docsUrl": "https://support.apple.com/en-us/119829",
"verificationMethod": "reverse-dns-and-ip-ranges",
"verificationUrl": "https://search.developer.apple.com/applebot.json",
"note": "Opt-out control only: Apple states Applebot-Extended does not crawl webpages, that pages disallowing it can still appear in search results, and that it only determines how data already crawled by Applebot may be used to train Apple's foundation models. The community snapshot carries no pattern for it, so this row's pattern was authored from Apple's page.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "common-crawl",
"operator": "Common Crawl",
"token": "CCBot",
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/robots-tokens?robotsTxtBehavior=user-triggered-may-ignore {
"data": [
{
"operatorId": "amazon",
"operator": "Amazon",
"token": "Amzn-User",
"robotsTxtLine": "User-agent: Amzn-User",
"purpose": "user-triggered",
"sendsRequests": true,
"robotsTxtBehavior": "user-triggered-may-ignore",
"fullUserAgentExample": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36",
"pattern": "Amzn-User",
"patternSource": "crawler-user-agents",
"docsUrl": "https://developer.amazon.com/amazonbot",
"verificationMethod": "published-ip-ranges",
"verificationUrl": "https://developer.amazon.com/amazonbot/live-ip-addresses/",
"note": "Amazon states this agent fetches live web information on a user's behalf and, because its actions can be user-initiated, may not follow all robots.txt directives. That is narrower than a blanket claim that it ignores robots.txt. Amazon also states Amzn-User does not crawl content for generative AI model training.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "meta",
"operator": "Meta",
"token": "meta-externalfetcher",
"robotsTxtLine": "User-agent: meta-externalfetcher",
"purpose": "user-triggered",
"sendsRequests": true,
"robotsTxtBehavior": "user-triggered-may-ignore",
"fullUserAgentExample": "meta-externalfetcher/1.1",
"pattern": "meta-externalfetcher\\/",
"patternSource": "crawler-user-agents",
"docsUrl": "https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers",
"verificationMethod": "none-published",
"verificationUrl": null,
"note": "Meta states this crawler fetches individual links at a user's request, supports agentic AI capabilities, and accordingly may bypass robots.txt rules.",
"verifiedOn": "2026-08-27"
},
{
"operatorId": "openai",
"operator": "OpenAI",
"token": "ChatGPT-User",
…
"generatedAt": "2026-09-15T18:48:07.000Z"
}
} /api/crawlers/robots-tokens?format=csv operatorId,operator,token,robotsTxtLine,purpose,sendsRequests,robotsTxtBehavior,fullUserAgentExample,pattern,patternSource,docsUrl,verificationMethod,verificationUrl,note,verifiedOn
amazon,Amazon,Amazonbot,User-agent: Amazonbot,general,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36",Amazonbot,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/ip-addresses/,"Amazon states this crawler ""is used to improve our products and services"" and that the data ""may be used to train Amazon AI models"". Purpose 'general' is the randomapi.dev classification of that broad wording; the possible training use stays visible here rather than being promoted into the crawler's sole purpose. Amazon also states its crawlers do not support the crawl-delay directive. Chrome/W.X.Y.Z is Amazon's own placeholder.",2026-08-27
amazon,Amazon,Amzn-SearchBot,User-agent: Amzn-SearchBot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36",Amzn-SearchBot,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/searchbot-ip-addresses/,"Amazon states this bot does not crawl content for generative AI model training, and that if robots.txt does not mention Amzn-SearchBot but allows other search bots it follows the directives given to those bots.",2026-08-27
amazon,Amazon,Amzn-User,User-agent: Amzn-User,user-triggered,true,user-triggered-may-ignore,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36",Amzn-User,crawler-user-agents,https://developer.amazon.com/amazonbot,published-ip-ranges,https://developer.amazon.com/amazonbot/live-ip-addresses/,"Amazon states this agent fetches live web information on a user's behalf and, because its actions can be user-initiated, may not follow all robots.txt directives. That is narrower than a blanket claim that it ignores robots.txt. Amazon also states Amzn-User does not crawl content for generative AI model training.",2026-08-27
anthropic,Anthropic,Claude-SearchBot,User-agent: Claude-SearchBot,search,true,obeys,,Claude-SearchBot,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states this bot analyses content to improve search result quality, and that disabling it prevents indexing for search and may reduce a site's visibility in search results.",2026-08-27
anthropic,Anthropic,Claude-User,User-agent: Claude-User,user-triggered,true,obeys,,Claude-User,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states this agent fetches pages when a person asks Claude a question, and lists it among the bots that honour robots.txt — unlike several other operators' user-triggered fetchers.",2026-08-27
anthropic,Anthropic,ClaudeBot,User-agent: ClaudeBot,ai-training,true,obeys,,[cC]laude[bB]ot,crawler-user-agents,https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler,published-ip-ranges,https://claude.com/crawling/bots.json,"Anthropic states that restricting ClaudeBot signals that a site's future material should be excluded from its AI model training datasets, and that its bots honour robots.txt and the non-standard Crawl-delay extension. Anthropic publishes tokens but no user-agent strings.",2026-08-27
apple,Apple,Applebot,User-agent: Applebot,search,true,obeys,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)",Applebot,crawler-user-agents,https://support.apple.com/en-us/119829,reverse-dns-and-ip-ranges,https://search.developer.apple.com/applebot.json,"Apple states that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot group, that Applebot does not follow crawl-delay, and that crawled data may also help train Apple's foundation models unless Applebot-Extended is disallowed. Traffic reverse-resolves under *.applebot.apple.com.",2026-08-27
apple,Apple,Applebot-Extended,User-agent: Applebot-Extended,ai-training,false,obeys,,Applebot-Extended,operator-documentation,https://support.apple.com/en-us/119829,reverse-dns-and-ip-ranges,https://search.developer.apple.com/applebot.json,"Opt-out control only: Apple states Applebot-Extended does not crawl webpages, that pages disallowing it can still appear in search results, and that it only determines how data already crawled by Applebot may be used to train Apple's foundation models. The community snapshot carries no pattern for it, so this row's pattern was authored from Apple's page.",2026-08-27
common-crawl,Common Crawl,CCBot,User-agent: CCBot,ai-training,true,obeys,CCBot/2.0 (https://commoncrawl.org/faq/),CCBot,crawler-user-agents,https://commoncrawl.org/ccbot,reverse-dns-and-ip-ranges,https://index.commoncrawl.org/ccbot.json,"Common Crawl's page describes CCBot only as producing ""an open repository of web crawl data that is universally accessible and analyzable by anyone"" — it does not mention AI training or characterize the repository as a training corpus. Purpose 'ai-training' is therefore a randomapi.dev classification for a crawler-use-preference list, not Common Crawl's wording. The page also warns that other crawlers falsely identify themselves as CCBot, and that real CCBot traffic reverse-resolves under crawl.commoncrawl.org (IPv4 only).",2026-08-27
google,Google,Google-CloudVertexBot,User-agent: Google-CloudVertexBot,site-owner-requested,true,obeys,,Google-CloudVertexBot,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google states preferences for this token affect crawls that site owners themselves request for building Vertex AI Agents, and have no effect on Google Search. Google publishes only the user-agent substring for it, not a full string, so fullUserAgentExample is null.",2026-08-27
google,Google,Google-Extended,User-agent: Google-Extended,ai-training,false,obeys,,Google-Extended,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Control token only: Google states it has no separate HTTP request user-agent string, that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect a site's inclusion in Google Search nor act as a ranking signal. Google also states the token governs two distinct things, not just one: whether crawled content may be used for training future generations of the Gemini models behind Gemini Apps and the Vertex AI API, AND grounding — supplying content from the Google Search index to the model at prompt time — in Gemini Apps and Grounding with Google Search on Vertex AI. A disallow preference applies to both named uses; it is not a separate crawler or a retroactive deletion mechanism. Nothing ever sends this string, so the verification link is for Google's actual crawler traffic.",2026-08-27
google,Google,GoogleOther,User-agent: GoogleOther,general,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36",GoogleOther,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google describes GoogleOther as the generic crawler that various product teams may use, and says crawling preferences for it do not affect any specific product.",2026-08-27
google,Google,Googlebot,User-agent: Googlebot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36",Googlebot\/,crawler-user-agents,https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers,reverse-dns-and-ip-ranges,https://developers.google.com/static/crawling/ipranges/common-crawlers.json,"Google publishes several Googlebot strings (this desktop one, a smartphone one and two shorter legacy forms); Chrome/W.X.Y.Z is Google's own version placeholder and is kept verbatim. Google says its common crawlers reverse-resolve under crawl-*.googlebot.com.",2026-08-27
meta,Meta,facebookexternalhit,User-agent: facebookexternalhit,link-preview,true,obeys-except-integrity-checks,facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php),facebookexternalhit,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content. Meta's page publishes no IP ranges or reverse-DNS domain, so there is no first-party verification link for any Meta crawler.",2026-08-27
meta,Meta,meta-externalads,User-agent: meta-externalads,advertising,true,obeys,meta-externalads/1.1,meta-externalads\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,Meta states this crawler supports use cases such as improving advertising and other business-related products.,2026-08-27
meta,Meta,meta-externalagent,User-agent: meta-externalagent,ai-training,true,obeys,meta-externalagent/1.1,meta-externalagent\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,Meta states this crawler is used for purposes such as training foundation AI models or improving products.,2026-08-27
meta,Meta,meta-externalfetcher,User-agent: meta-externalfetcher,user-triggered,true,user-triggered-may-ignore,meta-externalfetcher/1.1,meta-externalfetcher\/,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler fetches individual links at a user's request, supports agentic AI capabilities, and accordingly may bypass robots.txt rules.",2026-08-27
meta,Meta,meta-webindexer,User-agent: meta-webindexer,search,true,obeys,meta-webindexer/1.1,meta-webindexer,crawler-user-agents,https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers,none-published,,"Meta states this crawler ""navigates the web to improve Meta AI search result quality for users"" and that allowing it helps Meta cite and link the site in Meta AI's responses; purpose 'search' is the randomapi.dev classification of that wording rather than a category label Meta publishes.",2026-08-27
mistral,Mistral,MistralAI-Index,User-agent: MistralAI-Index,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)",MistralAI-Index,operator-documentation,https://docs.mistral.ai/robots,published-ip-ranges,https://mistral.ai/mistralai-index-ips.json,"Mistral states this agent crawls automatically for Mistral search indexing used to answer questions in Vibe, and that its content is not used for generative AI training. The community snapshot has no exact pattern for this published token, so this row's literal pattern was authored from Mistral's page.",2026-08-27
mistral,Mistral,MistralAI-Training,User-agent: MistralAI-Training,ai-training,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)",MistralAI-Training,operator-documentation,https://docs.mistral.ai/robots,none-published,,"Mistral states this crawler collects web content to help build datasets for training its generative AI models, can be disallowed in robots.txt, and is not used for search indexing or live Vibe queries. Mistral publishes no IP-address list for this token. The community snapshot has no exact pattern for it, so this row's literal pattern was authored from Mistral's page.",2026-08-27
mistral,Mistral,MistralAI-User,User-agent: MistralAI-User,user-triggered,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)",MistralAI-User,crawler-user-agents,https://docs.mistral.ai/robots,published-ip-ranges,https://mistral.ai/mistralai-user-ips.json,"Mistral states this agent supports user actions in Vibe, governs which sites those user-requested visits can access, and is not used for automatic web crawling or generative AI training.",2026-08-27
openai,OpenAI,ChatGPT-User,User-agent: ChatGPT-User,user-triggered,true,user-triggered-may-ignore,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot",ChatGPT-User,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/chatgpt-user.json,"OpenAI states that because these fetches are initiated by a user, robots.txt rules may not apply, and points at OAI-SearchBot for controlling automatic crawling.",2026-08-27
openai,OpenAI,GPTBot,User-agent: GPTBot,ai-training,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",GPTBot,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/gptbot.json,OpenAI states that disallowing GPTBot signals that a site's content should not be used to train its generative AI foundation models.,2026-08-27
openai,OpenAI,OAI-AdsBot,User-agent: OAI-AdsBot,advertising,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot",OAI-AdsBot,operator-documentation,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/adsbot.json,"OpenAI documents this token for validating the safety of pages submitted as ChatGPT ads, may use landing-page content to decide when an ad is relevant, and states the collected data is not used to train generative AI foundation models. OpenAI's advertiser guidance states that its crawlers respect robots.txt, crawling stops immediately when access is disallowed, and advertisers must allow OAI-AdsBot. No community pattern matches OpenAI's published string, so this row's pattern was authored from the crawler page.",2026-08-27
openai,OpenAI,OAI-SearchBot,User-agent: OAI-SearchBot,search,true,obeys,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot",OAI-SearchBot,crawler-user-agents,https://developers.openai.com/api/docs/bots,published-ip-ranges,https://openai.com/searchbot.json,"OpenAI recommends allowing OAI-SearchBot so a site can appear in ChatGPT search answers, and says a robots.txt change can take about 24 hours to take effect.",2026-08-27
perplexity,Perplexity,Perplexity-User,User-agent: Perplexity-User,user-triggered,true,user-triggered-may-ignore,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)",Perplexity-User,crawler-user-agents,https://docs.perplexity.ai/docs/resources/perplexity-crawlers,published-ip-ranges,https://www.perplexity.com/perplexity-user.json,"Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules.",2026-08-27
perplexity,Perplexity,PerplexityBot,User-agent: PerplexityBot,search,true,obeys,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",PerplexityBot\/,crawler-user-agents,https://docs.perplexity.ai/docs/resources/perplexity-crawlers,published-ip-ranges,https://www.perplexity.com/perplexitybot.json,"Perplexity states this bot surfaces and links sites in Perplexity search results and is not used to crawl content for AI foundation models. Its robots.txt policy says a blocked site will not have full or partial text indexed, but Perplexity may still retain the domain, headline and a brief factual summary. This is Perplexity's published policy, not independent verification of crawler behaviour.",2026-08-27
Two things, kept strictly apart because their provenance differs.
1502 user-agent patterns — the 1498 regular expressions in the crawler-user-agents project (MIT, pinned to one commit and verified by SHA-256), plus 4 patterns this repo authored for tokens an operator documents but that list does not carry. /api/crawlers lists them; /api/crawlers/detect?userAgent=… tells you what a string you have matches. Matching is case-sensitive and byte for byte: the upstream patterns encode their own case alternatives ([cC]laude[bB]ot), and 13 of them are ^-anchored, so googlebot/2.1 does not match and curl/8.7.1 does not match either. Both cases produce an explanatory warning rather than a silent no.
27 robots.txt tokens across 9 operators — /api/crawlers/robots-tokens is the curated half: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta, Amazon and Mistral, every row read off that operator's own documentation page (last checked 2026-08-27) and carrying that page's URL. It is a curated subset per operator, not a complete catalogue — Google alone documents eleven crawlers on its common-crawlers page and we curate four; each row's docsUrl is the full list. Two behavioural nuances come from additional first-party policy pages cited below: OpenAI's advertiser guidance for OAI-AdsBot and Perplexity's robots.txt policy for PerplexityBot.
operator, robotsToken, purpose, verification and note exist only on the 27 rows backed by a first-party operator page. purpose is always a randomapi.dev classification of the documented use, not a vendor-owned taxonomy; on the Amazonbot, CCBot and meta-webindexer rows the mapping is non-literal enough that each note labels it explicitly and preserves the operator's actual wording. On the other 1475 rows all five fields are null. We never derive a robots.txt token from a regular expression: ClaudeBot from [cC]laude[bB]ot would be a guess about capitalisation, and (^| )sentry\/ yields nothing at all. A wrong token in your robots.txt is worse than no token.Bytespider, get no curated row, and this page makes no claim about how they behave.exampleUserAgent is non-null on 118 of 1502 rows, by a published rule: the rows a curated token points at, every row the community list tags ai-crawler, and a committed high-traffic list, minus the control tokens that send no requests. That is not missing data — the field exists so you can paste a known-good string into /detect, which needs a well-chosen 118, not 1498. For anything else, paste a line from your own access log.Google-Extended and Applebot-Extended are control tokens only (sendsRequests: false). Nothing ever sends them, which is also why neither carries an example string — the community list does carry one for Google-Extended, and we drop it rather than publish a string no request sends.tags is the community project's classification, not the vendors' self-descriptions, and addedOn is when a pattern entered that list, not when the crawler started operating.robotsTxtBehavior is the operator's own published claim about itself. We do not test compliance or publish an accusation. Perplexity's policy says a block prevents indexing full or partial page text while still permitting retention of the domain, headline and a brief factual summary; the row reports that policy and explicitly does not claim independent verification.A User-Agent header is caller-supplied text. isKnownCrawler: false means "no pattern in this snapshot matched" — never "a human sent this", because a crawler impersonating a browser is exactly what a pattern list cannot see. Where an operator publishes IP ranges or reverse-DNS guidance, the curated row hands you that method and URL in verification; none-published is returned instead of a guess. We never fetch it, and there is no request-time network here at all.
/api/robots-txt/check is the other half of the workflow: list the tokens here, then paste a robots.txt and test the rule there. /api/reverse-dns builds the in-addr.arpa / ip6.arpa name you need for the PTR check. And do not confuse this with /api/user-agents, which generates plausible fake browser strings for fixtures — this endpoint carries a pinned list of real crawler patterns and never invents one.
/api/crawlers/robots-tokens?purpose=ai-training returns GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, MistralAI-Training with the exact User-agent: line for each. A disallow rule expresses a crawler-specific preference for future access or the named use; it does not retroactively delete content already collected or independently prove compliance. CCBot is a randomapi.dev classification, not Common Crawl's wording: Common Crawl describes an open repository of crawl data and does not characterize CCBot or that repository as an AI-training crawler or corpus. The other 6 labels follow uses stated on operator pages. Amazon's Amazonbot stays general: Amazon says it "is used to improve our products and services" and that its data "may be used to train Amazon AI models", so review that row separately if the possible use matters to you.
No. A User-Agent header is caller-supplied text, so anyone can send any string. Where an operator publishes IP ranges or reverse-DNS guidance, verification contains that method and URL; otherwise it explicitly returns none-published and null. This API never fetches the URL. Use /api/reverse-dns when the published method calls for a PTR check.
Training, search indexing and user-triggered fetching, all three documented by OpenAI: GPTBot crawls content that may train its models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches a page because a person asked. OpenAI states robots.txt rules may not apply to that last one. Anthropic and Perplexity publish the same kind of split.
No — it is a control token only. Google's own page states it "doesn't have a separate HTTP request user agent string", that crawling is done with existing Google user-agent strings and the token is used in a control capacity, and that it does not affect Search inclusion or ranking. Applebot-Extended works the same way, so both carry sendsRequests: false and no example string.
No. Nothing is fetched while your request is served. You supply the user-agent string to /api/crawlers/detect, and /api/robots-txt/check tests a robots.txt file you paste — neither route touches the network.
Test a URL path against a robots.txt you paste, for any crawler, and see the winning rule under RFC 9309 most-octets precedence.
Realistic, coherent browser user agents — filter by browser, OS and device; impossible combos like Safari-on-Windows get a clear 400, never a fake string.
Encode IPv4 and IPv6 addresses as canonical reverse-DNS PTR owner names, or decode complete in-addr.arpa and ip6.arpa names without a DNS query.
Random internet building blocks — valid IPv4/IPv6 addresses, locally-administered MACs, non-resolvable .test domains and URLs, slugs, ports and TLDs.
Complete pinned IANA HTTP field-name registry with status, Structured Field type, references, search and case-insensitive lookup.