Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
007ac9 Crawler
other
007ac9
observe
008
seo crawler
80legs
008
challenge
2checkout
other
Verifone
2checkout
allow
2ip bot
ai coding agent
2IP
observe
2ip.ru
other
2ip.ru
observe
360Monitoring
monitoring
360Monitoring
360monitoring
allow
360Spider
search engine
Qihoo 360
360Spider
allow
360Spider-Image
other
360Spider-Image
observe
360Spider-Video
other
360Spider-Video
observe
5emeRue
other
5emeRue
observe
5erue
other
5erue
observe
7Siters
other
7Siters
observe
A Patent Crawler
other
observe
A360-Search
ai assistant
Area360
A360-Search
observe
A6-Indexer
other
A6-Indexer
observe
aa
seo crawler
AgencyAnalytics
aa
challenge
AASA-Bot
ai assistant
Apple
AASA-Bot
observe
ABEvalBot
other
ABEvalBot
observe
Aboundex
other
Aboundex
observe
absearch-crawler
other
absearch-crawler
observe
AcademicBotRTU
other
AcademicBotRTU
observe
acapbot
other
acapbot
observe
Accessible Web Bot
other
Accessible Web
Accessible
allow
AccessibleWebBot
security
Accessible Web
AccessibleWebBot
allow
AccessStatus
monitoring
AccessStatus
AccessStatus
allow
acoonbot
other
acoonbot
observe
ActiveComply
ai assistant
ActiveComply
ActiveComply
observe
ActiveComply Bot
monitoring
ActiveComply
ActiveComply Bot
allow
Acunetix
security
Invicti Security
Acunetix
challenge
Acunetix Security Scanner
other
observe
Acunetix Web Vulnerability Scanner
other
observe
Adagio Bot
monitoring
Adagio
Adagiobot
allow
adbeat_bot
ai assistant
Adbeat
adbeat_bot
observe
adbeat-check-access-script
other
adbeat-check-access-script
observe
AdBot
other
AdBot
observe
AddSearchBot
seo crawler
Addsearch
AddSearchBot
challenge
AddThis
seo crawler
AddThis
AddThis
challenge
AddThis.com
ai assistant
Oracle
AddThis.com
observe
adequat
other
adequat
observe
adequat-systems
other
adequat-systems
observe
AdkernelTopicCrawler
ai assistant
AdKernel
AdkernelTopicCrawler
observe
admantx
ai assistant
Integral Ad Science
admantx
observe
AdmixerBot
ai assistant
Admixer
AdmixerBot
observe
ads-well-known-bot
other
ads-well-known-bot
observe
AdsBot-Google
ai assistant
AdsBot-Google
observe
AdsBot-Google-Mobile
ai assistant
AdsBot-Google-Mobile
observe
rDNS
adscanner
other
adscanner
observe
AdsTxtCrawler
seo crawler
OneTag Limited
AdsTxtCrawler
challenge
AdsTxtCrawler-CyberAgent
other
AdsTxtCrawler-CyberAgent
observe
Advailo Kuma
monitoring
Advailo
allow
AdvBot
other
AdvBot
observe
Adyen
other
Adyen
Adyen
allow
rDNS
AffsignalCrawler
ai assistant
Affsignal
AffsignalCrawler
observe
AfterShip Magento 2 Connector
other
AfterShip
allow
Agency Analytics Crawler
seo crawler
Agency Analytics
challenge
AgencyToolsPro Uptime Monitoring Bot
monitoring
AgencyToolsPro
AgencyToolsPro
allow
AgentReadinessScanner
ai coding agent
Cloudflare
AgentReadinessScanner
observe
AGI Agent
ai assistant
AGI, Inc
agi-agent-2
observe
signed
Ahrefs Site Audit
seo crawler
Ahrefs
AhrefsSiteAudit
challenge
AhrefsBot
seo crawler
Ahrefs
AhrefsBot
challenge