Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
007ac9 Crawler
other
007ac9
observe
2ip bot
ai coding agent
2IP
observe
2ip.ru
other
2ip.ru
observe
360Spider-Image
other
360Spider-Image
observe
360Spider-Video
other
360Spider-Video
observe
5emeRue
other
5emeRue
observe
5erue
other
5erue
observe
7Siters
other
7Siters
observe
A Patent Crawler
other
observe
A360-Search
ai assistant
Area360
A360-Search
observe
A6-Indexer
other
A6-Indexer
observe
AASA-Bot
ai assistant
Apple
AASA-Bot
observe
ABEvalBot
other
ABEvalBot
observe
Aboundex
other
Aboundex
observe
absearch-crawler
other
absearch-crawler
observe
AcademicBotRTU
other
AcademicBotRTU
observe
acapbot
other
acapbot
observe
acoonbot
other
acoonbot
observe
ActiveComply
ai assistant
ActiveComply
ActiveComply
observe
Acunetix Security Scanner
other
observe
Acunetix Web Vulnerability Scanner
other
observe
adbeat_bot
ai assistant
Adbeat
adbeat_bot
observe
adbeat-check-access-script
other
adbeat-check-access-script
observe
AdBot
other
AdBot
observe
AddThis.com
ai assistant
Oracle
AddThis.com
observe
adequat
other
adequat
observe
adequat-systems
other
adequat-systems
observe
AdkernelTopicCrawler
ai assistant
AdKernel
AdkernelTopicCrawler
observe
admantx
ai assistant
Integral Ad Science
admantx
observe
AdmixerBot
ai assistant
Admixer
AdmixerBot
observe
ads-well-known-bot
other
ads-well-known-bot
observe
AdsBot-Google
ai assistant
AdsBot-Google
observe
AdsBot-Google-Mobile
ai assistant
AdsBot-Google-Mobile
observe
rDNS
adscanner
other
adscanner
observe
AdsTxtCrawler-CyberAgent
other
AdsTxtCrawler-CyberAgent
observe
AdvBot
other
AdvBot
observe
AffsignalCrawler
ai assistant
Affsignal
AffsignalCrawler
observe
AgentReadinessScanner
ai coding agent
Cloudflare
AgentReadinessScanner
observe
AGI Agent
ai assistant
AGI, Inc
agi-agent-2
observe
signed
AI Search
ai crawler
Cloudflare
Cloudflare-AI-Search
observe
AI2Bot
ai scraper
Ai2
AI2Bot
observe
AI2Bot-DeepResearchEval
ai assistant
Ai2
AI2Bot-DeepResearchEval
observe
Ai2Bot-Dolma
ai scraper
Ai2
Ai2Bot-Dolma
observe
aiHitBot
ai assistant
aiHit
aiHitBot
observe
air.ai/scanning
other
air.ai/scanning
observe
AISearchBot
other
AISearchBot
observe
AIWebIndex
ai data provider
Lyrenth
AIWebIndex
observe
AIWebIndex-Agent
ai crawler
Lyrenth
AIWebIndex-Agent
observe
Alexabot
other
Alexabot
observe
Alexibot
other
Alexibot
observe
AllAfrica
ai assistant
AllAfrica Global Media
AllAfrica
observe
AllAfrica NewsBot
ai assistant
AllAfrica
observe
AlphaBot
other
AlphaBot
observe
Amazing-SearchBot
ai assistant
Amazing
Amazing-SearchBot
observe
AmazingTest-SearchBot
ai assistant
Amazing
AmazingTest-SearchBot
observe
Amazon Bedrock AgentCore Browser
other
Amazon
Amazon Bedrock AgentCore Browser
observe
Amazon Bedrock AgentCore Browser (AP Northeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-northeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP South)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-south-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast 1)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-1
observe
signed
Amazon Bedrock AgentCore Browser (AP Southeast)
ai assistant
Amazon
amazon-bedrock-agentcore-browser-ap-southeast-2
observe
signed