robots-txt

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared…

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

UNRATEDListed

Overview

Grade?

UNRATED

0 of 30 paid calls toward a letter

Price

$0.003 per call

Paid calls

none yet

delivery unknown until someone pays

More info

Seller?

0xb9f06d27…74b71acf

URL

https://api.vextorium.com/v1/robots-txt

MCP server on this host: com.vextorium.api/vextorium

Seen

first 12 hr ago · last 12 hr ago

1 free handshakes

Payment

Pays through

x402

Offered on?

eip155:10, Arc, Base USDC, Solana

Listed on?

Base, Solana

Paid calls

No paid calls yet.

Delivery is unknown until someone pays. A handshake only shows it asks for payment.

Verify it yourselfnpx teppi-check https://api.vextorium.com/v1/robots-txtcurl -s https://api.teppi.xyz/v1/trust/cap_01M42Q5MBJ42WSPZ0XC59E8204