Endpoint · extractionPOSTx402.forgemesh.io /robots-txt-ai-check
Fetches a domain's robots.txt and parses its Content-Signal and AI-preference directives (search, ai-input, ai-train) into structured JSON plus a plain-English summary of what's…
Read more
Fetches a domain's robots.txt and parses its Content-Signal and AI-preference directives (search, ai-input, ai-train) into structured JSON plus a plain-English summary of what's allowed — the "can this domain be crawled by an AI agent" check. Run before scraping content or building a RAG index, ahead of the industry's move toward stricter bot-gating defaults later this year.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.005 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://x402.forgemesh.io/robots-txt-ai-check
Seen
first 34 days ago · last 34 days ago
13 free handshakes
Payment
Pays through
x402 · MPP evm
Offered on?
Base USDC
Listed on?
Base
Handshakescomputed 27 days ago
0.784 from 13 free handshakes over 30 days · UNRATED ?
| Component | Weight | Measured | Lower bound ? | Adds | Short ? | Uncertain ? | |
|---|---|---|---|---|---|---|---|
| livenesscosts the most it answered at all |
0.60 | 1.00 | 0.771 | 0.462 | 0 | −0.138 | |
| latency p95 it answered as fast as its class |
0.20 | 0.61 | 0.606 | 0.121 | −0.079 | 0 | |
| price stability the price stayed where it was listed |
0.20 | 1.00 | 1.000 | 0.200 | 0 | 0 | |
| Composite | 1.00 | 0.784 | gap to 1.000 = 0.216 · 0.079 short · 0.138 uncertain | 0.784 | −0.079 | −0.138 |
Not measured: correctness · nobody paid; honesty · nobody paid; schema conformance · no answer had a published shape to check
BURN_INFREE_TIER_ONLY
Verify it yourself
npx teppi-check https://x402.forgemesh.io/robots-txt-ai-checkcurl -s https://api.teppi.xyz/v1/trust/cap_01M1A5Q9KEYCEBYHK9S8QVJ8AG