Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
SourceForge
MagpieRSS
Mailchimp
MailChimp
Mantis
Mantisbot
Manus
manus-bot
MapTheNetBot
Mastodon
Mastodon
MContextual
McontextualBot
Mediapartners-Google
Mediavine
MediavineMetadataParser
Medium
Mediumbot-MetaTagFetcher
Meta
meta-externalfetcher
Miniflux
Miniflux
Mistral AI
MistralAI-User
MixRank
MixrankBot
SavingsSites
MRLeadsBot
Muck Rack
MuckRackFeedParser
Nano Interactive
NanoInteractive
Nava Labs
Nekuda
nekuda-payment-executor
Netcraft
NetcraftSurveyAgent
netEstate
Neticle
Neticle
NetSeer
NetSeer
Netvibes
Netvibes
Neuwo
NeuwoBot
NewsBlur
NewsBlur
Newslit
Newslitbot
Marfeel
NewsRoom.BI
Nextcloud
NextCloud
VK
OdklBot
OMVION
OmvionLeadLake
Opengraph
OpenGraph.io
Osentix
osentix-crawler
Outbound Buddy
outboundbuddy-enrichment
Owler
Owler
SalesViewer
page-preview-tool
PagePeeker
PagePeeker
Domainsbot
Pandalytics
Panscient
panscient.com
PartnerOptimizer
PartnerOptimizer-bot
Payhawk
payhawk-invoice-fetching-bot
Perplexity
Perplexity-User
Phind
PhindBot
Piperic
PipericBot
Piqo
Piqo
PocketParser
Podchaser
PodchaserParser
Podimo
Podimo
Poggio
Poggio-Citations
PostGoo
PostGooBot
PostitLeadDiscovery
Keywords Everywhere
PoweredByBot
PressEngine
PressEngineBot
Productsup
productsup.io/crawler
PrognostiXon
PrognostiXonThinMiner
QA.tech
QATechBot
QUIC.cloud
qcbot
Quora
Quora-Bot
Quuu
QuuuBot