Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
A360-Search
ai assistant
Area360
A360-Search
observe
AASA-Bot
ai assistant
Apple
AASA-Bot
observe
adbeat_bot
ai assistant
Adbeat
adbeat_bot
observe
AddThis.com
ai assistant
Oracle
AddThis.com
observe
AdkernelTopicCrawler
ai assistant
AdKernel
AdkernelTopicCrawler
observe
admantx
ai assistant
Integral Ad Science
admantx
observe
AdmixerBot
ai assistant
Admixer
AdmixerBot
observe
AdpResearchBot
ai assistant
ADP
AdpResearchBot
observe
AdsBot-Google
ai assistant
AdsBot-Google
allow
AdsBot-Google-Mobile
ai assistant
AdsBot-Google-Mobile
allow
rDNS
AffsignalCrawler
ai assistant
Affsignal
AffsignalCrawler
observe
AGI Agent
ai assistant
AGI, Inc
agi-agent-2
observe
signed
AI2Bot-DeepResearchEval
ai assistant
Ai2
AI2Bot-DeepResearchEval
observe
aiHitBot
ai assistant
aiHit
aiHitBot
observe
AllAfrica NewsBot
ai assistant
AllAfrica
observe
allOrigins
ai assistant
allOrigins
observe
Amazing-SearchBot
ai assistant
Amazing
Amazing-SearchBot
observe
AmazingTest-SearchBot
ai assistant
Amazing
AmazingTest-SearchBot
observe
Amazon Bedrock AgentCore Browser (AP Northeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-northeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP South)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-south-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-2
observe
signed
Amazon Bedrock AgentCore Browser (EU Central 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-central-1
observe
signed
Amazon Bedrock AgentCore Browser (EU West 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-eu-west-1
observe
signed
Amazon Bedrock AgentCore Browser (US East 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east
observe
signed
Amazon Bedrock AgentCore Browser (US East 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-east-2
observe
signed
Amazon Bedrock AgentCore Browser (US West 2)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-us-west
observe
signed
Amazon Buy For Me
ai assistant
Amazon
AmazonBuyForMe
observe
Amazon-Advertising-ad-standards-bot
ai assistant
Amazon
Amazon-Advertising-ad-standards-bot
observe
amazon-QBusiness
ai assistant
Amazon
amazon-QBusiness
observe
AmazonProductDiscovery
ai assistant
Amazon
AmazonProductDiscovery
observe
Amzn-User
ai assistant
Amazon
Amzn-User
observe
AndersPinkBot
ai assistant
Anders Pink
AndersPinkBot
observe
ant-html
ai assistant
Kuaishou
ant-html
observe
Apify Website Content Crawler
ai assistant
Apify
ApifyWebsiteContentCrawler
observe
signed
ArchiveBot
ai assistant
Wikimedia
ArchiveBot
observe
ArenaUnfurlBot
ai assistant
Arena
ArenaUnfurlBot
observe
Assaybot
ai assistant
Index Exchange
Assaybot
observe
AudigentAdBot
ai assistant
Audigent
AudigentAdBot
observe
Automattic Analytics Crawler
ai assistant
WordPress
Automattic
observe
AutotouchCompanyCrawler
ai assistant
Autotouch
AutotouchCompanyCrawler
observe
AwarioBot
ai assistant
Awario
AwarioBot
observe
bitlybot
ai assistant
Bitly
bitlybot
observe
BitSightBot
ai assistant
Bitsight
BitSightBot
observe
Blackboard
ai assistant
Anthology
Blackboard
observe
Bloglines
ai assistant
Ask
Bloglines
observe
Bluesky
ai assistant
Bluesky
Bluesky
observe
BomboraBot
ai assistant
Bombora
BomboraBot
observe
BrandVerity
ai assistant
BrandVerity
BrandVerity
observe
Browserbase
ai assistant
Browserbase
Browserbase
observe
signed
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buttondown
ai assistant
Buttondown
Buttondown
observe
CandidOrgWebCheck
ai assistant
Candid
CandidOrgWebCheck
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
cartai-agentic-commerce-as-a-service
observe
signed
centryx-domain-mill
ai assistant
Centryx
centryx-domain-mill
observe
ChatGPT agent
ai assistant
OpenAI
ChatGPT
observe
signed
ChatGPT-User
ai assistant
OpenAI
ChatGPT-User
observe
CheckMarkNetwork
ai assistant
CheckMark Network
CheckMarkNetwork
observe