Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
471 results
Verifone
2checkout
360Monitoring
360monitoring
Qihoo 360
360Spider
Accessible Web
Accessible
Accessible Web
AccessibleWebBot
AccessStatus
AccessStatus
ActiveComply
ActiveComply Bot
Adagio
Adagiobot
Advailo
Adyen
Adyen
AfterShip
AgencyToolsPro
AgencyToolsPro
AI Trust Radar
AITrustRadar
Akismet
Akismet
Smartbear
AlertSite
Alexandria.org
Alexa
Amazon
Alexandria.org
AlexandriaOrgBot
Algolia
Algolia
Alibaba
AliyunSecBot
AllAfrica Global Media
AllAfrica
Appcast
AppcastParsingBot
Canadian Centre for Child Protection
Internet Archive
Archive-It
Area360
A360-Search
Arena.im
ArenaBot
Arquivo
Arquivo-web-crawler
Arquivo
Arquivo-web-crawler
capjamesg
artemis
Atlassian
atlassian-bot
RSSMicro
Atom
Audisto
Audisto
Automaton
Automaton
Baidu
Baidu-ADS
Baidu
Baiduspider
Baidu
Baiduspider-render
Bazqux
BazQux
Better Stack
Better
BigScoots
BigScootsMonitor
Binary Canary
binarycanary
Microsoft
BingPreview
Microsoft
bingbot
Microsoft
bingbot
Atlassian
Bitbucket-Webhooks
British Library
bl.uk_lddc_bot
Black Duck
Black
BlackVeil
BlackVeil-Security-Scanner
Blockaid
Blockaid
Blogtrottr
Blogtrottr
BlogVault
BlogVault
Bluesky
bne.es_bot
Bibliothèque nationale de France
bnf.fr_bot
Brynson
BrynsonComplianceBot
Buffer
BufferLinkPreviewBot
Bugs Nag
BugsNag
BuiltWith
BuiltWith
Cloudflare
bushbaby
Buttondown
Buttondown RSS-Feed-Parser
Catchpoint
Catchpoint