Skip to main content
Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked
Clear
All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

13 results · AI Data Provider

AIWebIndex ai data provider

Lyrenth

AIWebIndex
observe
ApifyBot ai data provider

Apify

ApifyBot
observe
ApifyWebsiteContentCrawler ai data provider

Apify

ApifyWebsiteContentCrawler
observe
Brightbot ai data provider

Bright Data

Brightbot
observe
CragCrawler ai data provider

CragSoftware

CragCrawler
observe
Crawlspace ai data provider

Crawlspace

Crawlspace
observe
Diffbot ai data provider

Diffbot

Diffbot
observe
ExaBot ai data provider

Exa

ExaBot
observe
HenkBot ai data provider

Valyu

HenkBot
observe
Mozilla-Tabstack ai data provider

Mozilla

Mozilla-Tabstack
observe
Querit-SearchBot ai data provider

Querit

Querit-SearchBot
observe
QueritBot ai data provider

Querit

QueritBot
observe
Terra ai data provider

Ceramic

Terra
observe