Endpoint · searchGETagent-data-tools.revmesh2074.workers.dev /robots ?url=https%3A%2F%2Fwww.google.com%2Fsearch
Robots.txt check.
Read more
Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.001 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch
MCP server on this host: io.github.Nutshell2000/agent-data-tools
Seen
first 23 days ago · last 4 days ago
29 free handshakes
Payment
Pays through
x402 · MPP evm, tempo
Offered on?
Polygon, Arbitrum, Base USDC, Solana, Tempo USDC.e
Listed on?
Polygon, Arbitrum, Base, Solana
Buy it through Teppi
- ?Pay on delivery
- 0.003000 USDC (0.001000 to the seller, 0.002000 fee)
- Where
- https://api.teppi.xyz/v1/via/cap_01M5VBS324A494JYKCWWR5TPJT
- ?Ask the price
curl -si https://api.teppi.xyz/v1/via/cap_01M5VBS324A494JYKCWWR5TPJT
About
- ?Description
- Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.
- URL
- https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch
- Method
- GET
- Class
- search
- ?Checkable to
- L1
- Endpoint id
- cap_01M5VBS324A494JYKCWWR5TPJT
- ?Badge
Reads live. UNRATED until paid calls earn a letter.
- Markdown
- [](https://teppi.xyz/endpoint/cap_01M5VBS324A494JYKCWWR5TPJT)
- HTML
- <a href="https://teppi.xyz/endpoint/cap_01M5VBS324A494JYKCWWR5TPJT"><img src="https://api.teppi.xyz/badge/cap_01M5VBS324A494JYKCWWR5TPJT.svg" alt="Teppi delivery record"></a>
- Image
- https://api.teppi.xyz/badge/cap_01M5VBS324A494JYKCWWR5TPJT.svg
Verify it yourself
npx teppi-check https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearchcurl -s https://api.teppi.xyz/v1/trust/cap_01M5VBS324A494JYKCWWR5TPJT