Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
AppInsights
other
AppInsights
observe
Apple App Site Association
other
Apple
observe
Applebot
ai crawler
Apple
Applebot
observe
rDNS
Applebot-Extended
ai scraper
Apple
Applebot-Extended
observe
rDNS
AppleNewsBot
other
AppleNewsBot
observe
Aqua_Products
other
Aqua_Products
observe
arabot
other
arabot
observe
Arachnid Shield API
security
Canadian Centre for Child Protection
allow
signed
Aranet-SearchBot
ai agent
Aranet-SearchBot
observe
Archive-It
archiver
Internet Archive
Archive-It
allow
ArchiveBot
ai assistant
Wikimedia
ArchiveBot
observe
Area 360
monitoring
Area360
A360-Search
allow
Arena Bot
social media
Arena.im
ArenaBot
allow
ArenaUnfurlBot
ai assistant
Arena
ArenaUnfurlBot
observe
Arquivo Web Crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Arquivo-web-crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Artemis Web Reader
social media
capjamesg
artemis
allow
Ask n read
other
observe
asknread.com
other
asknread.com
observe
AspiegelBot
other
AspiegelBot
observe
Assaybot
ai assistant
Index Exchange
Assaybot
observe
asterias
other
asterias
observe
AtlasBot
other
AtlasBot
observe
Atlassian (Rovo AI)
ai crawler
Atlassian
atlassian-bot
allow
atlassian-bot
ai crawler
Atlassian
atlassian-bot
observe
Atom Feed Robot
search engine
RSSMicro
Atom
allow
ATSScraper-discovery
other
ATSScraper-discovery
observe
Attracta
seo crawler
Attracta
Attracta
challenge
AtVowBot
other
AtVowBot
observe
AudigentAdBot
ai assistant
Audigent
AudigentAdBot
observe
Audisto Crawler
monitoring
Audisto
Audisto
allow
Augure
other
Augure
observe
auramundi
other
auramundi
observe
Authory
other
Authory
Authory
observe
Automaton
monitoring
Automaton
Automaton
allow
AutomatonHeadlessChrome
ai coding agent
Stack Moxie
AutomatonHeadlessChrome
observe
Automattic Analytics Crawler
ai assistant
WordPress
Automattic
observe
Automattic Feed Fetcher
other
observe
AwarioBot
ai assistant
Awario
AwarioBot
observe
AwarioRssBot
other
AwarioRssBot
observe
AwarioSmartBot
ai crawler
Awario
AwarioSmartBot
observe
awesomecrawler
other
awesomecrawler
observe
AzureAI-SearchBot
ai crawler
Microsoft
AzureAI-SearchBot
observe
B2B Bot
other
observe
b2w
other
b2w
observe
BackDoorBot
other
BackDoorBot
observe
BacklinkBot
other
BacklinkBot
observe
BacklinkCrawler
other
BacklinkCrawler
observe
BacklinksExtendedBot
other
BacklinksExtendedBot
observe
Bad Bots (mitchellkrogza)
other
1h4x.com
challenge
Baidu ADS Server Proxy
search engine
Baidu
Baidu-ADS
allow
Baidu-YunGuanCe
other
Baidu-YunGuanCe
observe
Baiduspider
search engine
Baidu
Baiduspider
allow
rDNS
Baiduspider-image
other
Baiduspider-image
observe
Baiduspider-news
other
Baiduspider-news
observe
Baiduspider-render
search engine
Baidu
Baiduspider-render
allow
Baiduspider-video
other
Baiduspider-video
observe
Barkrowler
seo crawler
Babbar
Barkrowler
challenge
Bazqux
social media
Bazqux
BazQux
allow
BDCbot
other
BDCbot
observe