Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

844 results · Other

Haloscan

HeadOnlyScraper other

HeadOnlyScraper

observe

HeadOnlyScraperV2 other

HelloWork Group

HetrixTools Uptime Monitoring Bot other

HIFI

HSTS preload bot other

Chromium

hstspreload-bot

observe

HTTP Banner Detection other

HubSpot Crawler other

HubSpot

HubSpot Connect 2.0 (http:

IMJ-CompanyPage-Scraper other

IMJ-CompanyPage-Scraper

observe

Impact.com Agent other

Instant Paper

Make

InterfaxScanBot other

InterfaxScanBot

observe

ip-web-crawler.com other

it2media-domain-crawler other

it2media-domain-crawler

kaikki.org-digital-archive other

kaikki.org-digital-archive

← Prev 6 / 15 Next →