Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

908 results · Other

intelium_bot other

intelium_bot

observe

InterfaxScanBot other

InterfaxScanBot

observe

ip-web-crawler.com other

IronFountain-Leads other

Iron Fountain

it2media-domain-crawler other

it2media-domain-crawler

kaikki.org-digital-archive other

kaikki.org-digital-archive

Keyword Density other

Known Agents Browser other

Known Agents

Landau-Media-Spider other

Landau-Media-Spider

LinkCheck by Siteimprove.com other

LinkextractorPro other

LinkextractorPro

observe

Linkhub-crawler other

← Prev 7 / 16 Next →