Test a URL path against a robots.txt you paste, for any crawler, and see the winning rule under RFC 9309 most-octets precedence.
Authoritative reference data or standards computation
Matches the path against the applying group's rules and returns the RFC 9309 verdict together with every competing rule and its octet length. Pass exactly one of url or path.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
robots
required
| string | example: User-agent: *
Disallow: /admin/
Allow: /admin/public/
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml | The robots.txt file text itself — this endpoint never fetches it. Newlines travel as %0A. Capped at 8000 bytes and 500 lines because it has to fit in a query string; larger input is rejected, never truncated. |
url | string | allowed: 1 – 2048 example: https://example.com/admin/public/logo.png | Absolute http(s) URL to test. Only the path and query are matched; the fragment is dropped and dot-segments are resolved, including their percent-encoded '%2e' spellings. A URL parser normalizes the rest too, so a literal space becomes '%20' and a backslash becomes '/'. Mutually exclusive with 'path'. |
path | string | allowed: 1 – 2048 example: /admin/settings | Path to test when you have no full URL. Must start with '/' and may include a query string; a fragment is dropped. Taken exactly as given — unlike 'url' it is never re-escaped by a URL parser, so a literal space stays a space instead of becoming '%20', and dot-segments are not resolved. Mutually exclusive with 'url'. |
userAgent | string | default: * allowed: 1 – 64 example: Googlebot | Crawler product token: '*' or letters, '-' and '_' only (RFC 9309 Section 2.2.1). Matched case-insensitively against the file's user-agent lines. |
robots string required The robots.txt file text itself — this endpoint never fetches it. Newlines travel as %0A. Capped at 8000 bytes and 500 lines because it has to fit in a query string; larger input is rejected, never truncated.
url string Absolute http(s) URL to test. Only the path and query are matched; the fragment is dropped and dot-segments are resolved, including their percent-encoded '%2e' spellings. A URL parser normalizes the rest too, so a literal space becomes '%20' and a backslash becomes '/'. Mutually exclusive with 'path'.
path string Path to test when you have no full URL. Must start with '/' and may include a query string; a fragment is dropped. Taken exactly as given — unlike 'url' it is never re-escaped by a URL parser, so a literal space stays a space instead of becoming '%20', and dot-segments are not resolved. Mutually exclusive with 'url'.
userAgent string Crawler product token: '*' or letters, '-' and '_' only (RFC 9309 Section 2.2.1). Matched case-insensitively against the file's user-agent lines.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
fields | list | example: userAgent,matchedGroup | Return only these fields (comma-separated). Mutually exclusive with 'exclude'. |
exclude | list | example: path | Return all fields except these (comma-separated). |
pretty | boolean | default: false example: true | Pretty-print the JSON response. |
unwrap | boolean | default: false example: true | Drop the envelope: return the raw array/object without data/meta wrapper. |
fields list Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
exclude list Return all fields except these (comma-separated).
pretty boolean Pretty-print the JSON response.
unwrap boolean Drop the envelope: return the raw array/object without data/meta wrapper.
| Field | Type | Description | Example |
|---|---|---|---|
userAgent | string | The product token that was tested, as supplied. Token matching itself is case-insensitive. | foobot |
matchedGroup nullable | string | Product token of the group that applied, as written in the file (the first one when several groups merge). '*' means the global group was the fallback; null means no group covers this crawler, so nothing is restricted. | foobot |
allowed | boolean | Whether the crawler may fetch the path. With no matching rule, the path is allowed. | false |
matchedRule nullable | object | The rule that decided the verdict: `type` (allow|disallow), `pattern` (canonical form compared against the path), `octetLength` (the precedence measure) and `lineNumber`. Null when no rule matched. | {"type":"disallow","pattern":"/example/page/disallowed.gif","octetLength":28,"lineNumber":3} |
competingRules | object[] | Every rule of the applying group(s) that matched the path, in the same shape as `matchedRule` and ordered by RFC 9309 precedence — most octets first, allow ahead of disallow on a tie — so the first entry is always the winner. | [{"type":"disallow","pattern":"/example/page/disallowed.gif","octetLength":28,"lineNumber":3},{"type":"allow","pattern":"/example/page/","octetLength":14,"lineNumber":2}] |
path | string | The path plus query string that was actually matched: the fragment removed, then the Section 2.2.2 comparison form applied (escapes of unreserved characters decoded, surviving escapes uppercased, non-ASCII UTF-8 percent-encoded). Reserved octets you send literally are compared literally. | /example/page/disallowed.gif |
userAgent string The product token that was tested, as supplied. Token matching itself is case-insensitive.
example: foobot
matchedGroup string nullable Product token of the group that applied, as written in the file (the first one when several groups merge). '*' means the global group was the fallback; null means no group covers this crawler, so nothing is restricted.
example: foobot
allowed boolean Whether the crawler may fetch the path. With no matching rule, the path is allowed.
example: false
matchedRule object nullable The rule that decided the verdict: `type` (allow|disallow), `pattern` (canonical form compared against the path), `octetLength` (the precedence measure) and `lineNumber`. Null when no rule matched.
example: {"type":"disallow","pattern":"/example/page/disallowed.gif","octetLength":28,"lineNumber":3}
competingRules object[] Every rule of the applying group(s) that matched the path, in the same shape as `matchedRule` and ordered by RFC 9309 precedence — most octets first, allow ahead of disallow on a tie — so the first entry is always the winner.
example: [{"type":"disallow","pattern":"/example/page/disallowed.gif","octetLength":28,"lineNumber":3},{"type":"allow","pattern":"/example/page/","octetLength":14,"lineNumber":2}]
path string The path plus query string that was actually matched: the fragment removed, then the Section 2.2.2 comparison form applied (escapes of unreserved characters decoded, surviving escapes uppercased, non-ASCII UTF-8 percent-encoded). Reserved octets you send literally are compared literally.
example: /example/page/disallowed.gif
/api/robots-txt/check?robots=User-agent:%20*%0ADisallow:%20/admin/%0AAllow:%20/admin/public/&path=/admin/settings {
"data": {
"userAgent": "*",
"matchedGroup": "*",
"allowed": false,
"matchedRule": {
"type": "disallow",
"pattern": "/admin/",
"octetLength": 7,
"lineNumber": 2
},
"competingRules": [
{
"type": "disallow",
"pattern": "/admin/",
"octetLength": 7,
"lineNumber": 2
}
],
"path": "/admin/settings"
},
"meta": {
"endpoint": "robots-txt",
"route": "/check",
"params": {
"robots": "User-agent: *\nDisallow: /admin/\nAllow: /admin/public/",
"path": "/admin/settings",
"userAgent": "*"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
} /api/robots-txt/check?robots=User-agent:%20foobot%0AAllow:%20/example/page/%0ADisallow:%20/example/page/disallowed.gif&userAgent=foobot&path=/example/page/disallowed.gif {
"data": {
"userAgent": "foobot",
"matchedGroup": "foobot",
"allowed": false,
"matchedRule": {
"type": "disallow",
"pattern": "/example/page/disallowed.gif",
"octetLength": 28,
"lineNumber": 3
},
"competingRules": [
{
"type": "disallow",
"pattern": "/example/page/disallowed.gif",
"octetLength": 28,
"lineNumber": 3
},
{
"type": "allow",
"pattern": "/example/page/",
"octetLength": 14,
"lineNumber": 2
}
],
"path": "/example/page/disallowed.gif"
},
"meta": {
"endpoint": "robots-txt",
"route": "/check",
"params": {
"robots": "User-agent: foobot\nAllow: /example/page/\nDisallow: /example/page/disallowed.gif",
"path": "/example/page/disallowed.gif",
"userAgent": "foobot"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
} /api/robots-txt/check?robots=User-agent:%20foobot%0AAllow:%20/example/page/%0ADisallow:%20/example/page/disallowed.gif&userAgent=foobot&path=/example/page/index.html {
"data": {
"userAgent": "foobot",
"matchedGroup": "foobot",
"allowed": true,
"matchedRule": {
"type": "allow",
"pattern": "/example/page/",
"octetLength": 14,
"lineNumber": 2
},
"competingRules": [
{
"type": "allow",
"pattern": "/example/page/",
"octetLength": 14,
"lineNumber": 2
}
],
"path": "/example/page/index.html"
},
"meta": {
"endpoint": "robots-txt",
"route": "/check",
"params": {
"robots": "User-agent: foobot\nAllow: /example/page/\nDisallow: /example/page/disallowed.gif",
"path": "/example/page/index.html",
"userAgent": "foobot"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
} /api/robots-txt/check?robots=User-agent:%20*%0AAllow:%20/docs/%0ADisallow:%20/*.pdf$&url=https://example.com/docs/manual.pdf {
"data": {
"userAgent": "*",
"matchedGroup": "*",
"allowed": false,
"matchedRule": {
"type": "disallow",
"pattern": "/*.pdf$",
"octetLength": 7,
"lineNumber": 3
},
"competingRules": [
{
"type": "disallow",
"pattern": "/*.pdf$",
"octetLength": 7,
"lineNumber": 3
},
{
"type": "allow",
"pattern": "/docs/",
"octetLength": 6,
"lineNumber": 2
}
],
"path": "/docs/manual.pdf"
},
"meta": {
"endpoint": "robots-txt",
"route": "/check",
"params": {
"robots": "User-agent: *\nAllow: /docs/\nDisallow: /*.pdf$",
"url": "https://example.com/docs/manual.pdf",
"userAgent": "*"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
}Reads the file exactly as /check does, but returns the structure instead of a verdict: every group with its product tokens and rules, plus sitemaps, non-standard records and the lines that could not be used.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
robots
required
| string | example: User-agent: *
Disallow: /admin/
Allow: /admin/public/
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml | The robots.txt file text itself — this endpoint never fetches it. Newlines travel as %0A. Capped at 8000 bytes and 500 lines because it has to fit in a query string; larger input is rejected, never truncated. |
robots string required The robots.txt file text itself — this endpoint never fetches it. Newlines travel as %0A. Capped at 8000 bytes and 500 lines because it has to fit in a query string; larger input is rejected, never truncated.
| Parameter | Type | Default & allowed | Description |
|---|---|---|---|
fields | list | example: groups,sitemaps | Return only these fields (comma-separated). Mutually exclusive with 'exclude'. |
exclude | list | example: lineCount | Return all fields except these (comma-separated). |
pretty | boolean | default: false example: true | Pretty-print the JSON response. |
unwrap | boolean | default: false example: true | Drop the envelope: return the raw array/object without data/meta wrapper. |
fields list Return only these fields (comma-separated). Mutually exclusive with 'exclude'.
exclude list Return all fields except these (comma-separated).
pretty boolean Pretty-print the JSON response.
unwrap boolean Drop the envelope: return the raw array/object without data/meta wrapper.
| Field | Type | Description | Example |
|---|---|---|---|
groups | object[] | Groups in file order: `userAgents` (the RFC 9309 product tokens that select the group) and `rules` ({ type, pattern, lineNumber }), each pattern as written in the file with only its comment and surrounding whitespace removed. | [{"userAgents":["*"],"rules":[{"type":"disallow","pattern":"/admin/","lineNumber":2},{"type":"allow","pattern":"/admin/public/","lineNumber":3}]}] |
sitemaps | string[] | Value of every Sitemap line, in file order. Sitemap is not defined by RFC 9309, and these URLs are never fetched or validated here. | ["https://example.com/sitemap.xml"] |
otherRecords | object[] | Records outside RFC 9309 such as crawl-delay, host or clean-param: { field, value, lineNumber }. Reported so you can see them, never evaluated by /check. | [{"field":"crawl-delay","value":"10","lineNumber":4}] |
invalidLines | object[] | Every line that could not be used, as { lineNumber, text, reason }: a missing ':' separator, an empty field name, an empty sitemap value, a user-agent value with no product token, or a rule that precedes the first user-agent line. The first five also appear in meta.warnings, on /check as well, so a dropped line is never silent. | [{"lineNumber":1,"text":"Disallow: /oops","reason":"rule precedes the first user-agent line, so RFC 9309 Section 2.2.2 requires it to be ignored"}] |
byteLength | integer | UTF-8 byte length of the text exactly as supplied, measured before parsing and including a byte order mark if you sent one. | 110 |
lineCount | integer | Number of lines, counting a single trailing line break as the end of the last line rather than a new one. | 5 |
groups object[] Groups in file order: `userAgents` (the RFC 9309 product tokens that select the group) and `rules` ({ type, pattern, lineNumber }), each pattern as written in the file with only its comment and surrounding whitespace removed.
example: [{"userAgents":["*"],"rules":[{"type":"disallow","pattern":"/admin/","lineNumber":2},{"type":"allow","pattern":"/admin/public/","lineNumber":3}]}]
sitemaps string[] Value of every Sitemap line, in file order. Sitemap is not defined by RFC 9309, and these URLs are never fetched or validated here.
example: ["https://example.com/sitemap.xml"]
otherRecords object[] Records outside RFC 9309 such as crawl-delay, host or clean-param: { field, value, lineNumber }. Reported so you can see them, never evaluated by /check.
example: [{"field":"crawl-delay","value":"10","lineNumber":4}]
invalidLines object[] Every line that could not be used, as { lineNumber, text, reason }: a missing ':' separator, an empty field name, an empty sitemap value, a user-agent value with no product token, or a rule that precedes the first user-agent line. The first five also appear in meta.warnings, on /check as well, so a dropped line is never silent.
example: [{"lineNumber":1,"text":"Disallow: /oops","reason":"rule precedes the first user-agent line, so RFC 9309 Section 2.2.2 requires it to be ignored"}]
byteLength integer UTF-8 byte length of the text exactly as supplied, measured before parsing and including a byte order mark if you sent one.
example: 110
lineCount integer Number of lines, counting a single trailing line break as the end of the last line rather than a new one.
example: 5
/api/robots-txt/parse?robots=User-agent:%20*%0ADisallow:%20/admin/%0AAllow:%20/admin/public/%0ACrawl-delay:%2010%0ASitemap:%20https://example.com/sitemap.xml {
"data": {
"groups": [
{
"userAgents": [
"*"
],
"rules": [
{
"type": "disallow",
"pattern": "/admin/",
"lineNumber": 2
},
{
"type": "allow",
"pattern": "/admin/public/",
"lineNumber": 3
}
]
}
],
"sitemaps": [
"https://example.com/sitemap.xml"
],
"otherRecords": [
{
"field": "crawl-delay",
"value": "10",
"lineNumber": 4
}
],
"invalidLines": [],
"byteLength": 110,
"lineCount": 5
},
"meta": {
"endpoint": "robots-txt",
"route": "/parse",
"params": {
"robots": "User-agent: *\nDisallow: /admin/\nAllow: /admin/public/\nCrawl-delay: 10\nSitemap: https://example.com/sitemap.xml"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
} /api/robots-txt/parse?robots=Disallow:%20/oops%0AUser-agent:%20Googlebot%0ADisallow%20/no-colon%0AAllow:%20/ {
"data": {
"groups": [
{
"userAgents": [
"Googlebot"
],
"rules": [
{
"type": "allow",
"pattern": "/",
"lineNumber": 4
}
]
}
],
"sitemaps": [],
"otherRecords": [],
"invalidLines": [
{
"lineNumber": 1,
"text": "Disallow: /oops",
"reason": "rule precedes the first user-agent line, so RFC 9309 Section 2.2.2 requires it to be ignored"
},
{
"lineNumber": 3,
"text": "Disallow /no-colon",
"reason": "no ':' separator — RFC 9309 records are '<field>: <value>'"
}
],
"byteLength": 65,
"lineCount": 4
},
"meta": {
"endpoint": "robots-txt",
"route": "/parse",
"params": {
"robots": "Disallow: /oops\nUser-agent: Googlebot\nDisallow /no-colon\nAllow: /"
},
"generatedAt": "2026-07-30T15:14:08.000Z",
…
]
}
} /api/robots-txt/parse?robots=User-agent:%20barbot%0AUser-agent:%20bazbot%0ADisallow:%20/private/%0AUser-agent:%20quxbot%0ADisallow:%20/ {
"data": {
"groups": [
{
"userAgents": [
"barbot",
"bazbot"
],
"rules": [
{
"type": "disallow",
"pattern": "/private/",
"lineNumber": 3
}
]
},
{
"userAgents": [
"quxbot"
],
"rules": [
{
"type": "disallow",
"pattern": "/",
"lineNumber": 5
}
]
}
],
"sitemaps": [],
"otherRecords": [],
"invalidLines": [],
"byteLength": 88,
"lineCount": 5
},
"meta": {
"endpoint": "robots-txt",
"route": "/parse",
"params": {
"robots": "User-agent: barbot\nUser-agent: bazbot\nDisallow: /private/\nUser-agent: quxbot\nDisallow: /"
},
"generatedAt": "2026-07-30T15:14:08.000Z"
}
}robots.txt verdicts computed from a file you paste in — nothing is ever fetched, which is exactly why this works from CI against a committed robots.txt, and on a file you have not published yet.
/check answers "may this crawler fetch this path, and which line decided it?". Every allow/disallow rule in the applying group is matched against the path and the rule with the most octets in its path pattern wins — "The most specific match is the match that has the most octets" (Section 2.2.2) — with an exact tie going to allow. competingRules returns every rule that matched, ordered by that same precedence, so the decision is auditable instead of asserted.
Group selection follows Section 2.2.1: product tokens are matched case-insensitively, every group whose token equals your userAgent is merged into one, the * group applies only when no specific group matches, and when neither exists nothing is restricted. Per Section 2.2.3, * stands for any sequence of characters and a trailing $ anchors the end of the path; matching starts at the first octet of the path, so an unanchored pattern only has to match a prefix. The query string is part of what is matched (Disallow: /*? behaves as authors expect) and the fragment is dropped, because it never reaches a server.
/parse gives the same reading of the file without a verdict: groups (every rule carrying its line number), sitemaps, otherRecords and invalidLines.
Scope limits, stated up front.
noindex for that.Crawl-delay, Host and friends are reported, never evaluated — RFC 9309 does not define them. Neither does it define Sitemap, which it only names as an example of another record; those values are listed and never fetched.robots is limited to 8000 bytes and 500 lines — far below the RFC's 500 KiB parsing minimum. Over the cap you get a 400 stating the real byte and line counts; the file is never truncated, because a truncated robots.txt produces a wrong verdict.Allow: /a/%62%61%7A against /a/baz. Note also that HTTP query parsing decodes your input once, so a path that genuinely contains %2F must be sent double-encoded as %252F.?baz=https://foo.bar → ?baz=https%3A%2F%2Ffoo.bar). That step belongs to whoever builds the URI, not to the comparison — the same figure's first row leaves the reserved ? and = raw, and escaping reserved octets inside a pattern would destroy * and $. So /foo/bar?baz=https://foo.bar is compared exactly as written, which is what the reference parser does; encode it yourself if you want it matched encoded.: separator and one of the names RFC 9309 defines. The reference C++ parser is deliberately lenient here — it accepts a space instead of a missing colon (Disallow /admin) and a table of frequent misspellings (useragent, dissallow, site-map) — and this endpoint does not, so a file Googlebot obeys can be a file we ignore. Every line we drop is reported: /parse lists it in invalidLines or otherRecords, and both routes warn about it in meta.warnings, naming the line number, so a stricter reading never comes back as a silent "crawlable".url and path normalize differently, on purpose. url goes through a URL parser, so dot-segments (including %2e spellings) are resolved and the path and query are re-escaped the way a crawler's own URL library would do it: /a b becomes /a%20b and a backslash becomes /. path is taken byte for byte — the same treatment rule patterns get — so Disallow: /a b matches path=/a b but not url=http://e.com/a b. Pass url whenever you have one; a path carrying an octet a URL parser would rewrite is warned about.- and _, and a User-agent: value is truncated at the first character outside that set (so MJ12bot is the token MJ), with a warning — the same thing the reference parser does. A userAgent argument that is not a product token is a 400 rather than a guess.No — you paste the file into the robots parameter and we never fetch anything, so a draft or a file sitting in your repository tests exactly like a live one.
The one with the most octets in its path pattern (RFC 9309, Section 2.2.2), and on an exact tie the allow wins. /check returns every matching rule in competingRules with its octetLength, ordered by that rule.
It covers the part that went away when Google retired that tester in December 2023: testing an arbitrary path against arbitrary rules. It is an RFC 9309 evaluator, not a simulation of Googlebot.
No. robots.txt governs crawling, not access or indexing — a disallowed URL is still publicly reachable and can still surface in search results if other pages link to it.
/parse reports it under otherRecords with its line number, but /check never evaluates it: crawl-delay is not part of RFC 9309, and verdicts come only from allow and disallow rules.
Product tokens match case-insensitively; all groups whose token equals your userAgent are merged, the * group is used only when no specific group matches, and with neither present every path is allowed.
The current IANA delegated TLD ASCII list — search all labels, filter punycode IDNs, or look up one TLD with an honest 404.
Parse absolute URLs or resolve relative references with the WHATWG URL algorithm — canonical hosts, ordered query pairs and no credential leakage.
Realistic, coherent browser user agents — filter by browser, OS and device; impossible combos like Safari-on-Windows get a clear 400, never a fake string.
Generate safe DNS record fixtures with type-correct RDATA and coherent SRV RRsets, using reserved domains and documentation addresses only.
The complete IANA HTTP status code registry with practical descriptions, defining RFCs and a retryable flag — list by category or look up a single code.
Known crawler and AI-bot user agents: the robots.txt token each operator publishes, and what a user-agent string you supply actually matches.