Endpoints: 76,849MCP servers: 32,722Payout addresses: 3,825Paid calls: 6,098Letters: 83Defects: 3,891counted 4 min ago
teppi

Endpoint · searchGETfreetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage

Check whether a URL may be crawled per the site's robots.txt for a given user-agent.
Read moreCheck whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.
UNRATEDListed

Overview

Grade?
UNRATED
0 of 30 paid calls toward a letter
Price
$0.003 per call
Paid calls
none yet
delivery unknown until someone pays

More info

URL
https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage
Seen
first 17 days ago · last 4 days ago
22 free handshakes

Payment

Pays through
x402
Offered on?
Base USDC
Listed on?
Base

Buy it through Teppi

?Pay on delivery
0.005000 USDC (0.003000 to the seller, 0.002000 fee)
Where
https://api.teppi.xyz/v1/via/cap_01M6AT47ANFEK4JYBF22D1FGR8
?Ask the price
curl -si https://api.teppi.xyz/v1/via/cap_01M6AT47ANFEK4JYBF22D1FGR8

About

?Description
Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.
URL
https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpage
Method
GET
Class
search
?Checkable to
L1
Endpoint id
cap_01M6AT47ANFEK4JYBF22D1FGR8

?Badge

Reads live. UNRATED until paid calls earn a letter.

Markdown
[![Teppi](https://api.teppi.xyz/badge/cap_01M6AT47ANFEK4JYBF22D1FGR8.svg)](https://teppi.xyz/endpoint/cap_01M6AT47ANFEK4JYBF22D1FGR8)
HTML
<a href="https://teppi.xyz/endpoint/cap_01M6AT47ANFEK4JYBF22D1FGR8"><img src="https://api.teppi.xyz/badge/cap_01M6AT47ANFEK4JYBF22D1FGR8.svg" alt="Teppi delivery record"></a>
Image
https://api.teppi.xyz/badge/cap_01M6AT47ANFEK4JYBF22D1FGR8.svg
Verify it yourselfnpx teppi-check https://freetier-agent-api.pages.dev/api/robots-check?ua=MyAgent&url=https%3A%2F%2Fexample.com%2Fsome%2Fpagecurl -s https://api.teppi.xyz/v1/trust/cap_01M6AT47ANFEK4JYBF22D1FGR8