Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
antibot
other
antibot
observe
AntiSpam-Agent
other
AntiSpam-Agent
observe
AnyEvent
other
AnyEvent
observe
Apercite
other
Apercite
observe
Apify Website Content Crawler
ai assistant
Apify
ApifyWebsiteContentCrawler
observe
signed
ApifyBot
ai data provider
Apify
ApifyBot
observe
ApifyWebsiteContentCrawler
ai data provider
Apify
ApifyWebsiteContentCrawler
observe
AppcastBot
social media
Appcast
AppcastParsingBot
allow
AppInsights
other
AppInsights
observe
Apple App Site Association
other
Apple
observe
Applebot
ai crawler
Apple
Applebot
observe
rDNS
Applebot-Extended
ai scraper
Apple
Applebot-Extended
observe
rDNS
AppleNewsBot
other
AppleNewsBot
observe
Aqua_Products
other
Aqua_Products
observe
arabot
other
arabot
observe
Arachnid Shield API
security
Canadian Centre for Child Protection
allow
signed
Aranet-SearchBot
ai agent
Aranet-SearchBot
observe
Archive-It
archiver
Internet Archive
Archive-It
allow
ArchiveBot
ai assistant
Wikimedia
ArchiveBot
observe
Area 360
monitoring
Area360
A360-Search
allow
Arena Bot
social media
Arena.im
ArenaBot
allow
ArenaUnfurlBot
ai assistant
Arena
ArenaUnfurlBot
observe
Arquivo Web Crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Arquivo-web-crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Artemis Web Reader
social media
capjamesg
artemis
allow
artsdata-crawler
ai data provider
Culture Creates
artsdata-crawler
observe
Ask n read
other
observe
asknread.com
other
asknread.com
observe
AskyBot
other
AskyBot
observe
AspiegelBot
other
AspiegelBot
observe
Assaybot
ai assistant
Index Exchange
Assaybot
observe
asterias
other
asterias
observe
AtlasBot
other
AtlasBot
observe
Atlassian (Rovo AI)
ai crawler
Atlassian
atlassian-bot
allow
atlassian-bot
ai crawler
Atlassian
atlassian-bot
observe
Atom Feed Robot
search engine
RSSMicro
Atom
allow
ATSScraper-discovery
other
ATSScraper-discovery
observe
Attracta
seo crawler
Attracta
Attracta
challenge
AtVowBot
other
AtVowBot
observe
AudigentAdBot
ai assistant
Audigent
AudigentAdBot
observe
Audisto Crawler
monitoring
Audisto
Audisto
allow
Augure
other
Augure
observe
auramundi
other
auramundi
observe
Authory
other
Authory
Authory
observe
Automaton
monitoring
Automaton
Automaton
allow
AutomatonHeadlessChrome
ai coding agent
Stack Moxie
AutomatonHeadlessChrome
observe
Automattic Analytics Crawler
ai assistant
WordPress
Automattic
observe
Automattic Feed Fetcher
other
observe
AutotouchCompanyCrawler
ai assistant
Autotouch
AutotouchCompanyCrawler
observe
AwarioBot
ai assistant
Awario
AwarioBot
observe
AwarioRssBot
other
AwarioRssBot
observe
AwarioSmartBot
ai crawler
Awario
AwarioSmartBot
observe
awesomecrawler
other
awesomecrawler
observe
AzureAI-SearchBot
ai crawler
Microsoft
AzureAI-SearchBot
observe
B2B Bot
other
observe
b2w
other
b2w
observe
BackDoorBot
other
BackDoorBot
observe
BacklinkBot
other
BacklinkBot
observe
BacklinkCrawler
other
BacklinkCrawler
observe
BacklinksExtendedBot
other
BacklinksExtendedBot
observe