Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
Aqua_Products
other
Aqua_Products
observe
arabot
other
arabot
observe
Arachnid Shield API
security
Canadian Centre for Child Protection
allow
signed
Aranet-SearchBot
ai agent
Aranet-SearchBot
observe
Archive-It
archiver
Internet Archive
Archive-It
allow
ArchiveBot
ai assistant
Wikimedia
ArchiveBot
observe
Area 360
monitoring
Area360
allow
Arena Bot
social media
Arena.im
allow
ArenaUnfurlBot
ai assistant
Arena
ArenaUnfurlBot
observe
Arquivo Web Crawler
archiver
Arquivo
allow
Arquivo-web-crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Artemis Web Reader
social media
capjamesg
artemis
allow
Ask
other
Ask
observe
asknread.com
other
asknread.com
observe
AspiegelBot
other
AspiegelBot
observe
Assaybot
ai assistant
Index Exchange
Assaybot
observe
asterias
other
asterias
observe
AtlasBot
other
AtlasBot
observe
Atlassian (Rovo AI)
ai crawler
Atlassian
allow
atlassian-bot
ai crawler
Atlassian
atlassian-bot
observe
Atom
search engine
RSSMicro
Atom
allow
ATSScraper-discovery
other
ATSScraper-discovery
observe
Attracta
seo crawler
Attracta
Attracta
challenge
AtVowBot
other
AtVowBot
observe
AudigentAdBot
ai assistant
Audigent
AudigentAdBot
observe
Audisto Crawler
monitoring
Audisto
Audisto
allow
Augure
other
Augure
observe
auramundi
other
auramundi
observe
Authory
other
Authory
Authory
observe
Automaton
monitoring
Automaton
Automaton
allow
AutomatonHeadlessChrome
ai coding agent
Stack Moxie
AutomatonHeadlessChrome
observe
Automattic
ai assistant
WordPress
Automattic
observe
AwarioBot
ai assistant
Awario
AwarioBot
observe
AwarioRssBot
other
AwarioRssBot
observe
AwarioSmartBot
ai crawler
Awario
Awario
observe
awesomecrawler
other
awesomecrawler
observe
AzureAI-SearchBot
ai crawler
Microsoft
AzureAI-SearchBot
observe
B2B
other
B2B
observe
b2w
other
b2w
observe
BackDoorBot
other
BackDoorBot
observe
BacklinkBot
other
BacklinkBot
observe
BacklinkCrawler
other
BacklinkCrawler
observe
BacklinksExtendedBot
other
BacklinksExtendedBot
observe
Bad Bots (mitchellkrogza)
other
challenge
Baidu ADS Server Proxy
search engine
Baidu
Baidu-ADS
allow
Baidu-YunGuanCe
other
Baidu-YunGuanCe
observe
Baiduspider
search engine
Baidu
Baiduspider
allow
Baiduspider-image
other
Baiduspider-image
observe
Baiduspider-news
other
Baiduspider-news
observe
Baiduspider-render
search engine
Baidu
Baiduspider-render
allow
Baiduspider-video
other
Baiduspider-video
observe
Barkrowler
seo crawler
Babbar
Barkrowler
challenge
Bazqux
social media
Bazqux
BazQux
allow
BDCbot
other
BDCbot
observe
bedrockbot
ai scraper
Amazon
bedrockbot
observe
BehloolBot
other
BehloolBot
observe
BestChange Bot
seo crawler
BestChange
BestChange
challenge
betaBot
other
betaBot
observe
Better
ai coding agent
Better Stack
Better
observe
Better Uptime
monitoring
Better Stack
Better
allow