MCP serverio.github.taylorsmithgg/crawl-census

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

UNRATEDActivestreamable-httpcrawlcensus.com

Overview

Score?

UNRATED 0.788

of what a free look can see, on 30 looks

Looks

last 15 hr ago

Tools

URL

crawlcensus.com/mcp

streamable-http

Says it is

crawl-census 1.0.0

protocol 2025-06-18

In the record since

32 days ago

0median 0.606 · this server 0.788 · highest on record 0.8561

The tools this server lists, read out of the definition it returned
Tool	Schema
agent_profile What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network e	input · no output
census_facts Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready	input · no output
census_stats Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.	input · no output
crawl_preflight Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is r	input · no output
scan_site Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Re	input · no output
site_report Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.	input · no output
submit_domains Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurabl	input · no output

Verify it yourselfnpx teppi-check https://crawlcensus.com/mcpcurl -s https://api.teppi.xyz/v1/trust/mcp/mcs_01M1FZ2NEGVWSFS6EZYQA4CJR4