robots?domain=string

Fetch and parse robots.txt for a domain.

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.

UNRATEDListedProse in the URL

Overview

Grade?

UNRATED

0 of 30 paid calls toward a letter

Price

$0.005 per call

Paid calls

none yet

delivery unknown until someone pays

More info

Seller?

0x4466d4a8…d6dd125e

URL

https://api.x402node.dev/web/robots?domain=string

Seen

first 35 days ago · last 4 days ago

63 free handshakes

Payment

Pays through

x402

Offered on?

Base USDC, Solana

Listed on?

Base, Solana

Paid calls

No paid calls yet.

Delivery is unknown until someone pays. A handshake only shows it asks for payment.

Paid for on the same host1 of 439 endpoints at api.x402node.dev

Other endpoints on api.x402node.dev that were paid for
Endpoint	Grade	Paid calls	Last result	Last seen
GET /aviation/airport-status	UNRATED	1	delivered	15 hr ago

All 439 endpoints on this host

Verify it yourselfnpx teppi-check https://api.x402node.dev/web/robots?domain=stringcurl -s https://api.teppi.xyz/v1/trust/cap_01M1A5QEQGKME86BC53EAQXPSM