Rankelle

Crawler

The automated reader search engines use to find pages.

A crawler follows links, fetches pages and reads their markup. Everything an audit tool knows about a site starts here, which is why a site that blocks crawlers is unmeasurable rather than healthy.

Ours identifies itself as Rankelle-Audit, obeys robots.txt, rate-limits per host, and discards page text after hashing the fields that matter.

Try it on your own site, free

No card, and the free tier does not expire — the first verdict takes about six weeks to arrive.