Endpoint · extractionGETnetintel.dev /robots-txt/analyze ?domain=example.com
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked…
Read more
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.01 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://netintel.dev/robots-txt/analyze?domain=example.com
Seen
first 35 days ago · last 14 hr ago
64 free handshakes
Payment
Pays through
x402
Offered on?
Base USDC, Solana
Listed on?
Base, Solana
Buy it through Teppi
- ?Pay on delivery
- 0.012000 USDC (0.010000 to the seller, 0.002000 fee)
- Where
- https://api.teppi.xyz/v1/via/cap_01M1A59M5GWWXMXJK609Y01EHD
- ?Ask the price
curl -si https://api.teppi.xyz/v1/via/cap_01M1A59M5GWWXMXJK609Y01EHD
About
- ?Description
- Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
- URL
- https://netintel.dev/robots-txt/analyze?domain=example.com
- Method
- GET
- Class
- extraction
- ?Checkable to
- L1
- Endpoint id
- cap_01M1A59M5GWWXMXJK609Y01EHD
- ?Badge
Reads live. UNRATED until paid calls earn a letter.
- Markdown
- [](https://teppi.xyz/endpoint/cap_01M1A59M5GWWXMXJK609Y01EHD)
- HTML
- <a href="https://teppi.xyz/endpoint/cap_01M1A59M5GWWXMXJK609Y01EHD"><img src="https://api.teppi.xyz/badge/cap_01M1A59M5GWWXMXJK609Y01EHD.svg" alt="Teppi delivery record"></a>
- Image
- https://api.teppi.xyz/badge/cap_01M1A59M5GWWXMXJK609Y01EHD.svg
Verify it yourself
npx teppi-check https://netintel.dev/robots-txt/analyze?domain=example.comcurl -s https://api.teppi.xyz/v1/trust/cap_01M1A59M5GWWXMXJK609Y01EHD