Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
Amazon Bedrock AgentCore Browser (EU Central 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-central-1
observe
signed
Amazon Bedrock AgentCore Browser (EU West 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-west-1
observe
signed
Amazon Bedrock AgentCore Browser (US East 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east
observe
signed
Amazon Bedrock AgentCore Browser (US East 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east-2
observe
signed
Amazon Bedrock AgentCore Browser (US West 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-west
observe
signed
Amazon Buy For Me
ai assistant
Amazon
AmazonBuyForMe
observe
Amazon Kendra
ai crawler
Amazon
amazon-kendra
observe
Amazon-Advertising-ad-standards-bot
ai assistant
Amazon
Amazon-Advertising-ad-standards-bot
observe
amazon-QBusiness
ai assistant
Amazon
amazon-QBusiness
observe
Amazonbot
ai crawler
Amazon
Amazonbot
observe
rDNS
AmazonBuyForMe
ai agent
Amazon
AmazonBuyForMe
observe
AmazonProductDiscovery
ai assistant
Amazon
AmazonProductDiscovery
observe
AmiSoftware
other
AmiSoftware
observe
Amzn-SearchBot
ai crawler
Amazon
Amzn-SearchBot
observe
Amzn-User
ai assistant
Amazon
Amzn-User
observe
Anchor Browser
ai crawler
Anchor
Anchor
observe
signed
AndersPinkBot
ai assistant
Anders Pink
AndersPinkBot
observe
Anomura
ai crawler
Direqt
Anomura
observe
anthropic-ai
ai agent
Anthropic
anthropic-ai
observe
antibot
other
antibot
observe
AntiSpam-Agent
other
AntiSpam-Agent
observe
AnyEvent
other
AnyEvent
observe
Apercite
other
Apercite
observe
Apify Website Content Crawler
ai assistant
Apify
ApifyWebsiteContentCrawler
observe
signed
ApifyBot
ai data provider
Apify
ApifyBot
observe
ApifyWebsiteContentCrawler
ai data provider
Apify
ApifyWebsiteContentCrawler
observe
AppInsights
other
AppInsights
observe
Apple App Site Association
other
Apple
observe
Applebot
ai crawler
Apple
Applebot
observe
rDNS
Applebot-Extended
ai scraper
Apple
Applebot-Extended
observe
rDNS
AppleNewsBot
other
AppleNewsBot
observe
Aqua_Products
other
Aqua_Products
observe
arabot
other
arabot
observe
Aranet-SearchBot
ai agent
Aranet-SearchBot
observe
ArchiveBot
ai assistant
Wikimedia
ArchiveBot
observe
ArenaUnfurlBot
ai assistant
Arena
ArenaUnfurlBot
observe
Ask n read
other
observe
asknread.com
other
asknread.com
observe
AspiegelBot
other
AspiegelBot
observe
Assaybot
ai assistant
Index Exchange
Assaybot
observe
asterias
other
asterias
observe
AtlasBot
other
AtlasBot
observe
atlassian-bot
ai crawler
Atlassian
atlassian-bot
observe
ATSScraper-discovery
other
ATSScraper-discovery
observe
AtVowBot
other
AtVowBot
observe
AudigentAdBot
ai assistant
Audigent
AudigentAdBot
observe
Augure
other
Augure
observe
auramundi
other
auramundi
observe
Authory
other
Authory
Authory
observe
AutomatonHeadlessChrome
ai coding agent
Stack Moxie
AutomatonHeadlessChrome
observe
Automattic Analytics Crawler
ai assistant
WordPress
Automattic
observe
Automattic Feed Fetcher
other
observe
AwarioBot
ai assistant
Awario
AwarioBot
observe
AwarioRssBot
other
AwarioRssBot
observe
AwarioSmartBot
ai crawler
Awario
AwarioSmartBot
observe
awesomecrawler
other
awesomecrawler
observe
AzureAI-SearchBot
ai crawler
Microsoft
AzureAI-SearchBot
observe
B2B Bot
other
observe
b2w
other
b2w
observe
BackDoorBot
other
BackDoorBot
observe