Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
AI2Bot-DeepResearchEval
ai assistant
Ai2
AI2Bot-DeepResearchEval
observe
Ai2Bot-Dolma
ai scraper
Ai2
Ai2Bot-Dolma
observe
aiHitBot
ai assistant
aiHit
aiHitBot
observe
air.ai/scanning
other
air.ai/scanning
observe
AISearchBot
other
AISearchBot
observe
AITrustRadar
security
AI Trust Radar
AITrustRadar
allow
AIWebIndex
ai data provider
Lyrenth
AIWebIndex
observe
Akismet
other
Akismet
allow
Alertsite
ai coding agent
SmartBear
Alertsite
observe
Alertsite by Smartbear
monitoring
Smartbear
AlertSite
allow
Alexa
search engine
Alexandria.org
Alexa
allow
Alexabot
other
Alexabot
observe
AlexandriaOrgBot
search engine
Alexandria.org
AlexandriaOrgBot
allow
Alexibot
other
Alexibot
observe
Algolia
search engine
Algolia
Algolia
allow
AliyunSecBot
security
Alibaba
AliyunSecBot
allow
All Africa Crawler
search engine
AllAfrica Global Media
AllAfrica
allow
Alli AI Bot
seo crawler
Alli AI
challenge
AlphaBot
other
AlphaBot
observe
Amazing-SearchBot
ai assistant
Amazing
Amazing-SearchBot
observe
AmazingTest-SearchBot
ai assistant
Amazing
AmazingTest-SearchBot
observe
Amazon
other
Amazon
Amazon
observe
Amazon AdBot
seo crawler
Amazon
AmazonAdBot
challenge
Amazon Bedrock AgentCore Browser (AP Northeast)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (AP South)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast 1)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (EU Central 1)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (EU West 1)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (US East 1)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (US East 2)
ai assistant
Amazon
observe
signed
Amazon Bedrock AgentCore Browser (US West 2)
ai assistant
Amazon
observe
signed
Amazon Buy For Me
ai assistant
Amazon
AmazonBuyForMe
observe
Amazon Contxbot
seo crawler
Amazon
challenge
Amazon Kendra
ai crawler
Amazon
amazon-kendra
observe
Amazon-Advertising-ad-standards-bot
ai assistant
Amazon
Amazon-Advertising-ad-standards-bot
observe
Amazon-Bedrock-AgentCore-Browser
browser automation
Amazon Web Services
observe
amazon-QBusiness
ai assistant
Amazon
amazon-QBusiness
observe
Amazonbot
ai crawler
Amazon
Amazonbot
observe
AmazonProductDiscovery
ai assistant
Amazon
AmazonProductDiscovery
observe
AmiSoftware
other
AmiSoftware
observe
Amzn-SearchBot
ai crawler
Amazon
Amzn-SearchBot
observe
Amzn-User
ai assistant
Amazon
Amzn-User
observe
Anchor Browser
ai crawler
Anchor
Anchor
observe
signed
AndersPinkBot
ai assistant
Anders Pink
AndersPinkBot
observe
Anomura
ai crawler
Direqt
Anomura
observe
anthropic-ai
ai agent
Anthropic
anthropic-ai
observe
antibot
other
antibot
observe
AntiSpam-Agent
other
AntiSpam-Agent
observe
AnyEvent
other
AnyEvent
observe
Apercite
other
Apercite
observe
Apify Website Content Crawler
ai assistant
Apify
observe
signed
ApifyBot
ai data provider
Apify
ApifyBot
observe
ApifyWebsiteContentCrawler
ai data provider
Apify
ApifyWebsiteContentCrawler
observe
AppcastBot
social media
Appcast
allow
AppInsights
other
AppInsights
observe
Apple App Site Association
other
Apple
observe
Applebot
ai crawler
Apple
Applebot
observe
Applebot-Extended
ai scraper
Apple
Applebot-Extended
observe
AppleNewsBot
other
AppleNewsBot
observe