Endpoint · extractionPOSTweb.openverbs.com /v1/robots
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule…
Read more
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.006 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://web.openverbs.com/v1/robots
Seen
first 44 days ago · last 4 days ago
49 free handshakes
Payment
Pays through
x402 · MPP evm, tempo
Offered on?
Base USDC, Tempo 0x8335…2913
Listed on?
Base
Buy it through Teppi
- ?Pay on delivery
- 0.008000 USDC (0.006000 to the seller, 0.002000 fee)
- Where
- https://api.teppi.xyz/v1/via/cap_01M459KP0KBJNP56F2NX67A97W
- ?Ask the price
curl -si -X POST https://api.teppi.xyz/v1/via/cap_01M459KP0KBJNP56F2NX67A97W -H 'content-type: application/json' -d '{}'
About
- ?Description
- Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.
- URL
- https://web.openverbs.com/v1/robots
- Method
- POST
- Class
- extraction
- ?Checkable to
- L1
- Endpoint id
- cap_01M459KP0KBJNP56F2NX67A97W
- ?Badge
Reads live. UNRATED until paid calls earn a letter.
- Markdown
- [](https://teppi.xyz/endpoint/cap_01M459KP0KBJNP56F2NX67A97W)
- HTML
- <a href="https://teppi.xyz/endpoint/cap_01M459KP0KBJNP56F2NX67A97W"><img src="https://api.teppi.xyz/badge/cap_01M459KP0KBJNP56F2NX67A97W.svg" alt="Teppi delivery record"></a>
- Image
- https://api.teppi.xyz/badge/cap_01M459KP0KBJNP56F2NX67A97W.svg
Verify it yourself
npx teppi-check https://web.openverbs.com/v1/robotscurl -s https://api.teppi.xyz/v1/trust/cap_01M459KP0KBJNP56F2NX67A97W