Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
22 results · AI Data Provider
AgentDataBot
ai data provider
AgentData
AgentDataBot
observe
AIWebIndex
ai data provider
Lyrenth
AIWebIndex
observe
ApifyBot
ai data provider
Apify
ApifyBot
observe
ApifyWebsiteContentCrawler
ai data provider
Apify
ApifyWebsiteContentCrawler
observe
artsdata-crawler
ai data provider
Culture Creates
artsdata-crawler
observe
BixelBot
ai data provider
Bixel
BixelBot
observe
Brightbot
ai data provider
Bright Data
Brightbot
observe
CloudflareBrowserRenderingCrawler
ai data provider
Cloudflare
CloudflareBrowserRenderingCrawler
observe
CragCrawler
ai data provider
CragSoftware
CragCrawler
observe
Crawlspace
ai data provider
Crawlspace
Crawlspace
observe
DesearchBot
ai data provider
Desearch
DesearchBot
observe
Diffbot
ai data provider
Diffbot
Diffbot
observe
ExaBot
ai data provider
Exa
ExaBot
observe
HenkBot
ai data provider
Valyu
HenkBot
observe
Keenable-User
ai data provider
Keenable
Keenable-User
observe
KeenableBot
ai data provider
Keenable
KeenableBot
observe
Mozilla-Tabstack
ai data provider
Mozilla
Mozilla-Tabstack
observe
Querit-SearchBot
ai data provider
Querit
Querit-SearchBot
observe
QueritBot
ai data provider
Querit
QueritBot
observe
ScryBot
ai data provider
Unflatten
ScryBot
observe
SofyaBot
ai data provider
Sofya
SofyaBot
observe
TalarionFrontierCrawler
ai data provider
Talarion
TalarionFrontierCrawler
observe