Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
471 results
Macrobondbot
Macrobondbot
Peak Labs
MagiBot
Mail Russia
Mail.RU_Bot
Direct Support / Website Managed
MainWP
Make
ManageWP
ManageWP
Marginalia
search.marginalia.nu
Mars Flag
Mars
Mavifinds
Mavifinds
Mediaboard
mediaboardbot
Medialogia
MedialogiaBot
Meta
meta-webindexer
Metorik
Metorik
Microsoft
MicrosoftPreview
MirrorWeb
mirrorweb
Uniqoders Technologies SL
Mojeek
MojeekBot
Mollie B.V.
Monitage
Monitage
Monitis
monitis
MonitoRSS
MonitoRSS
Acquia
Acquia
MonSpark
MonSpark
Montastic
MontasticMonitor
Montonio
MontonioWebhooks
MRG Web Services srl
MRGbot
Microsoft
msnbot
Microsoft
msnbot-media
Muck Rack
MuckRack
Mwmbl
mwmbl
Neeva
Neevabot
NestDaddy
NestDaddyBot
Netarkivet
netarkivindsamling
Netcraft
Netcraft
Netcraft
Netumo
netumo
Netvigie
NETVIGIE
New Relic
NewRelicbot
NewsBank
NewsBank
NewsBlur
NewsBlur Favicon Fetcher - 1692 subscribers - https:
NewsNowUK
NewsNow
NiceCrawler
Nicecrawler
NixStats
NixStatsMonitoringBot
Nmap
Nmap
Nodeping
NodePing
Noibu
Noibu
Nooshub
Nooshub
Helpfeel
Notabot
Notion
NotionAutomation
Observer
Observer
ODIN
Odin
Oh Dear
OhDear
OKX
OKX-dolphin-crawler
Omnisend
Omnisend
OneTrust
OneTrust
OneTrust, LLC.
Onetrust
OnlineOrNot
OnlineOrNot.com_bot_1.0_(https:
Opengraph
OpenRSS
OpenRSS
Orlo
Orlo-LinkPreview