Endpoint · extractionGETapi.x402node.dev /web/robots ?domain=string
Fetch and parse robots.txt for a domain.
Read more
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.005 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://api.x402node.dev/web/robots?domain=string
Seen
first 2 months ago · last 4 days ago
112 free handshakes
Payment
Pays through
x402
Offered on?
Base USDC, Solana
Listed on?
Base, Solana
Score, from free handshakes onlyweights 2026.09.6
UN
RATED
RATED
0.834 liveness verified, no letter
To a letter · 30 paid calls
0 of 30
Held back by sample size, not by the seller · gap to 1.000 is 0.166: 0.120 uncertain ? + 0.046 short ? · 0.066 to A
| Component | Weight | Measured | Lower bound ? | Adds | Short ? | Uncertain ? | |
|---|---|---|---|---|---|---|---|
| livenesscosts the most it answered at all |
0.60 | 1.00 | 0.802 | 0.481 | 0 | −0.119 | |
| latency p95 it answered as fast as its class |
0.20 | 0.77 | 0.764 | 0.153 | −0.046 | −0.001 | |
| price stability the price stayed where it was listed |
0.20 | 1.00 | 1.000 | 0.200 | 0 | 0 | |
| Composite | 1.00 | 0.834 | gap to 1.000 = 0.166 · 0.046 short · 0.120 uncertain | 0.834 | −0.046 | −0.120 |
Not measured: correctness · nobody paid; honesty · nobody paid; schema conformance · no answer had a published shape to check
The multiplication, written out
0.600 × 0.802 liveness
+ 0.200 × 0.764 latency p95
+ 0.200 × 1.000 price stability
= 0.834 → UNRATED: 31 of 30 paid calls weights 2026.09.6 · 31 samples · seed cap_01M1A5QEQGKME86BC53EAQXPSM|2026-10-18T00:14:24.229Z|2026.09.6
Verify it yourself
npx teppi-check https://api.x402node.dev/web/robots?domain=stringcurl -s https://api.teppi.xyz/v1/trust/cap_01M1A5QEQGKME86BC53EAQXPSM