Endpoint · searchGETfreetier-agent-api.pages.dev /api/robots-check ?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage
Check whether a URL may be crawled per the site's robots.txt for a given user-agent.
Read more
Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.Overview
Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.003 per call
Paid calls
none yet
delivery unknown until someone pays
More info
Seller?
URL
https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage
MCP server on this host: dev.pages.freetier-agent-api/freetier-agent-api
Seen
first 17 days ago · last 4 days ago
22 free handshakes
Payment
Pays through
x402
Offered on?
Base USDC
Listed on?
Base
Buy it through Teppi
- ?Pay on delivery
- 0.005000 USDC (0.003000 to the seller, 0.002000 fee)
- Where
- https://api.teppi.xyz/v1/via/cap_01M6AT47ANFEK4JYBF22D1FGR8
- ?Ask the price
curl -si https://api.teppi.xyz/v1/via/cap_01M6AT47ANFEK4JYBF22D1FGR8
About
- ?Description
- Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.
- URL
- https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage
- Method
- GET
- Class
- search
- ?Checkable to
- L1
- Endpoint id
- cap_01M6AT47ANFEK4JYBF22D1FGR8
- ?Badge
Reads live. UNRATED until paid calls earn a letter.
- Markdown
- [](https://teppi.xyz/endpoint/cap_01M6AT47ANFEK4JYBF22D1FGR8)
- HTML
- <a href="https://teppi.xyz/endpoint/cap_01M6AT47ANFEK4JYBF22D1FGR8"><img src="https://api.teppi.xyz/badge/cap_01M6AT47ANFEK4JYBF22D1FGR8.svg" alt="Teppi delivery record"></a>
- Image
- https://api.teppi.xyz/badge/cap_01M6AT47ANFEK4JYBF22D1FGR8.svg
Verify it yourself
npx teppi-check https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpagecurl -s https://api.teppi.xyz/v1/trust/cap_01M6AT47ANFEK4JYBF22D1FGR8