Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
AI Search
ai crawler
Cloudflare
Cloudflare-AI-Search
observe
AI2Bot
ai scraper
Ai2
AI2Bot
observe
AI2Bot-DeepResearchEval
ai assistant
Ai2
AI2Bot-DeepResearchEval
observe
Ai2Bot-Dolma
ai scraper
Ai2
Ai2Bot-Dolma
observe
aiHitBot
ai assistant
aiHit
aiHitBot
observe
air.ai/scanning
other
air.ai/scanning
observe
AISearchBot
other
AISearchBot
observe
AITrustRadar
security
AI Trust Radar
AITrustRadar
allow
AIWebIndex
ai data provider
Lyrenth
AIWebIndex
observe
AIWebIndex-Agent
ai crawler
Lyrenth
AIWebIndex-Agent
observe
Akismet
other
Akismet
Akismet
allow
Alertsite by Smartbear
monitoring
Smartbear
AlertSite
allow
Alexa Archive
search engine
Alexandria.org
Alexa
allow
alexa site audit
search engine
Amazon
allow
Alexabot
other
Alexabot
observe
AlexandriaOrgBot
search engine
Alexandria.org
AlexandriaOrgBot
allow
Alexibot
other
Alexibot
observe
Algolia
search engine
Algolia
Algolia
allow
AliyunSecBot
security
Alibaba
AliyunSecBot
allow
All Africa Crawler
search engine
AllAfrica Global Media
AllAfrica
allow
AllAfrica
ai assistant
AllAfrica Global Media
AllAfrica
observe
AllAfrica NewsBot
ai assistant
AllAfrica
observe
Alli AI Bot
seo crawler
Alli AI
challenge
AlphaBot
other
AlphaBot
observe
Amazing-SearchBot
ai assistant
Amazing
Amazing-SearchBot
observe
AmazingTest-SearchBot
ai assistant
Amazing
AmazingTest-SearchBot
observe
Amazon AdBot
seo crawler
Amazon
AmazonAdBot
challenge
rDNS
Amazon Bedrock AgentCore Browser
other
Amazon
Amazon Bedrock AgentCore Browser
observe
Amazon Bedrock AgentCore Browser (AP Northeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-northeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP South)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-south-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-2
observe
signed
Amazon Bedrock AgentCore Browser (EU Central 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-central-1
observe
signed
Amazon Bedrock AgentCore Browser (EU West 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-west-1
observe
signed
Amazon Bedrock AgentCore Browser (US East 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east
observe
signed
Amazon Bedrock AgentCore Browser (US East 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east-2
observe
signed
Amazon Bedrock AgentCore Browser (US West 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-west
observe
signed
Amazon Buy For Me
ai assistant
Amazon
AmazonBuyForMe
observe
Amazon Contxbot
seo crawler
Amazon
contxbot
challenge
Amazon Kendra
ai crawler
Amazon
amazon-kendra
observe
Amazon-Advertising-ad-standards-bot
ai assistant
Amazon
Amazon-Advertising-ad-standards-bot
observe
amazon-QBusiness
ai assistant
Amazon
amazon-QBusiness
observe
Amazonbot
ai crawler
Amazon
Amazonbot
observe
rDNS
AmazonBuyForMe
ai agent
Amazon
AmazonBuyForMe
observe
AmazonProductDiscovery
ai assistant
Amazon
AmazonProductDiscovery
observe
AmiSoftware
other
AmiSoftware
observe
Amzn-SearchBot
ai crawler
Amazon
Amzn-SearchBot
observe
Amzn-User
ai assistant
Amazon
Amzn-User
observe
Anchor Browser
ai crawler
Anchor
Anchor
observe
signed
AndersPinkBot
ai assistant
Anders Pink
AndersPinkBot
observe
Anomura
ai crawler
Direqt
Anomura
observe
anthropic-ai
ai agent
Anthropic
anthropic-ai
observe
antibot
other
antibot
observe
AntiSpam-Agent
other
AntiSpam-Agent
observe
AnyEvent
other
AnyEvent
observe
Apercite
other
Apercite
observe
Apify Website Content Crawler
ai assistant
Apify
ApifyWebsiteContentCrawler
observe
signed
ApifyBot
ai data provider
Apify
ApifyBot
observe
ApifyWebsiteContentCrawler
ai data provider
Apify
ApifyWebsiteContentCrawler
observe
AppcastBot
social media
Appcast
AppcastParsingBot
allow