Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
SourceForge
MagpieRSS
Magus
Mailchimp
MailChimp
Mail Russia
Mail.RU_Bot
Direct Support / Website Managed
MainWP
Make
Make.com
make.com
Make Merry
MakeMerryBot
ManageWP
ManageWP
ManageWP
ManageWP-UptimeMonitoring
Mantis
Mantisbot
Manus
manus-bot
Butterfly Effect
Manus-User
mappydata
MapTheNetBot
Marginalia
search.marginalia.nu
marketgoo
MarketGoo
Mars Flag
Mars
Mastodon
Mastodon
MastodonInstances
Mata
MatchboxBot
MatchorySearch
MathPicDatasetCrawler
Mattermost-Bot
MauiBot
Mavifinds
Mavifinds
SEOptimer
MBCrawler
MContextual
McontextualBot
Mediaboard
mediaboardbot
Medialogia
MedialogiaBot
MediaMonitoringBot
MediaMonitoringBot
Mediapartners-Google
Determ
Determ
Mediavine
Mediavine Medatada Parser
Mediavine
MediavineMetadataParser
Medium
Mediumbot-MetaTagFetcher
MegaIndex
MegaIndex
MegaIndex.ru
MelonMesa
CentComBot
Meltawer
Meltwater
MeltwaterNews
memorybot
mention
Stripe
MerchantSecurityScanner
MeridianBot
Meta
meta-externalads
Meta
meta-externalagent
Meta
meta-externalfetcher
Meta
meta-webindexer
MetaCrawler
MetadataScraper
MetaInspector
METAJob
MetaJobBot
MetaURI
Metorik
Metorik
MGID
MgidBot
Reflection AI
micro-crawl
Microsoft
MicrosoftPreview