Endpoint · extractionGETapi.x402node.dev /web/robots ?domain=string
Fetch and parse robots.txt for a domain.
Read more
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.005 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://api.x402node.dev/web/robots?domain=string
Seen
first 35 days ago · last 4 days ago
63 free handshakes
Payment
Pays through
x402
Offered on?
Base USDC, Solana
Listed on?
Base, Solana
About
- ?Description
- Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.
- URL
- https://api.x402node.dev/web/robots?domain=string
- Method
- GET
- Class
- extraction
- ?Checkable to
- L1
- Endpoint id
- cap_01M1A5QEQGKME86BC53EAQXPSM
- ?Badge
Reads live. UNRATED until paid calls earn a letter.
- Markdown
- [](https://teppi.xyz/endpoint/cap_01M1A5QEQGKME86BC53EAQXPSM)
- HTML
- <a href="https://teppi.xyz/endpoint/cap_01M1A5QEQGKME86BC53EAQXPSM"><img src="https://api.teppi.xyz/badge/cap_01M1A5QEQGKME86BC53EAQXPSM.svg" alt="Teppi delivery record"></a>
- Image
- https://api.teppi.xyz/badge/cap_01M1A5QEQGKME86BC53EAQXPSM.svg
Verify it yourself
npx teppi-check https://api.x402node.dev/web/robots?domain=stringcurl -s https://api.teppi.xyz/v1/trust/cap_01M1A5QEQGKME86BC53EAQXPSM