Endpoints: 76,849MCP servers: 32,731Payout addresses: 3,825Paid calls: 6,171Letters: 84Defects: 3,897counted 2 min ago
teppi

Endpoint · searchGETagent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch

Robots.txt check.
Read moreRobots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.
UNRATEDListed

Overview

Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.001 per call
Paid calls
none yet
delivery unknown until someone pays

More info

URL
https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch
Seen
first 23 days ago · last 4 days ago
29 free handshakes

Payment

Pays through
x402 · MPP evm, tempo
Offered on?
Polygon, Arbitrum, Base USDC, Solana, Tempo USDC.e
Listed on?
Polygon, Arbitrum, Base, Solana

Buy it through Teppi

?Pay on delivery
0.003000 USDC (0.001000 to the seller, 0.002000 fee)
Where
https://api.teppi.xyz/v1/via/cap_01M5VBS324A494JYKCWWR5TPJT
?Ask the price
curl -si https://api.teppi.xyz/v1/via/cap_01M5VBS324A494JYKCWWR5TPJT

About

?Description
Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.
URL
https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch
Method
GET
Class
search
?Checkable to
L1
Endpoint id
cap_01M5VBS324A494JYKCWWR5TPJT

?Badge

Reads live. UNRATED until paid calls earn a letter.

Markdown
[![Teppi](https://api.teppi.xyz/badge/cap_01M5VBS324A494JYKCWWR5TPJT.svg)](https://teppi.xyz/endpoint/cap_01M5VBS324A494JYKCWWR5TPJT)
HTML
<a href="https://teppi.xyz/endpoint/cap_01M5VBS324A494JYKCWWR5TPJT"><img src="https://api.teppi.xyz/badge/cap_01M5VBS324A494JYKCWWR5TPJT.svg" alt="Teppi delivery record"></a>
Image
https://api.teppi.xyz/badge/cap_01M5VBS324A494JYKCWWR5TPJT.svg
Verify it yourselfnpx teppi-check https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearchcurl -s https://api.teppi.xyz/v1/trust/cap_01M5VBS324A494JYKCWWR5TPJT