Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
Make.com
ai crawler
Make.com
observe
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
ManageWP
other
ManageWP
allow
Mantisbot
ai assistant
Mantis
Mantisbot
observe
Manus Bot
ai assistant
Manus
observe
Manus-User
browser automation
Butterfly Effect
Manus-User
observe
mappydata
other
mappydata
observe
Marginalia
search engine
Marginalia
Marginalia
allow
Marginalia Search
search engine
Marginalia
allow
marketgoo
seo crawler
marketgoo
MarketGoo
challenge
Mars Finder
search engine
Mars Flag
Mars
allow
Mastodon
ai assistant
Mastodon
Mastodon
observe
MastodonInstances
other
MastodonInstances
observe
Mata
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MauiBot
other
MauiBot
observe
Mavifinds Bot
monitoring
Mavifinds
Mavifinds
allow
MBCrawler
seo crawler
SEOptimer
MBCrawler
challenge
McontextualBot
ai assistant
MContextual
McontextualBot
observe
mediaboardbot
monitoring
Mediaboard
mediaboardbot
allow
Medialogia Bot
monitoring
Medialogia
MedialogiaBot
allow
MediaMonitoringBot
seo crawler
MediaMonitoringBot
MediaMonitoringBot
challenge
Mediapartners-Google
ai assistant
Mediapartners-Google
observe
Mediatoolkitbot
seo crawler
Determ
Determ
challenge
Mediavine
ai assistant
Mediavine
Mediavine
observe
Mediavine Medatada Parser
seo crawler
Mediavine
challenge
MediavineMetadataParser
ai assistant
Mediavine
MediavineMetadataParser
observe
Mediumbot-MetaTagFetcher
ai assistant
Medium
Mediumbot-MetaTagFetcher
observe
MegaIndex
other
MegaIndex
observe
MegaIndex.ru
seo crawler
MegaIndex
MegaIndex.ru
challenge
MelonMesa Bot
other
MelonMesa
observe
Meltawer
other
Meltawer
observe
Meltwater
other
Meltwater
observe
MeltwaterNews
other
MeltwaterNews
observe
memorybot
other
memorybot
observe
mention
other
mention
observe
Meta-ExternalAds
seo crawler
Meta
meta-externalads
challenge
Meta-ExternalAgent
ai crawler
Meta
meta-externalagent
observe
Meta-ExternalFetcher
ai assistant
Meta
meta-externalfetcher
observe
Meta-WebIndexer
social media
Meta
meta-webindexer
allow
MetadataScraper
other
MetadataScraper
observe
MetaInspector
seo crawler
MetaInspector
challenge
MetaJobBot
seo crawler
METAJob
MetaJobBot
challenge
MetaURI
other
MetaURI
observe
Metorik
other
Metorik
allow
MgidBot
seo crawler
MGID
MgidBot
challenge
micro-crawl
ai scraper
Reflection AI
micro-crawl
observe
Microsoft
ai assistant
Microsoft
Microsoft
observe
Microsoft Preview
social media
Microsoft
MicrosoftPreview
allow
MIIxpc
other
MIIxpc
observe
mindUpBot
other
mindUpBot
observe
minicrawler
other
minicrawler
observe
Miniflux
ai assistant
Miniflux
Miniflux
observe
Minoru's
other
Minoru's
observe
mirrorweb
archiver
MirrorWeb
mirrorweb
allow
MirrorWebCrawler
seo crawler
MirrorWeb Ltd
MirrorWebCrawler
challenge
Missinglettr Bot
seo crawler
Missinglettr
MissinglettrBot
challenge
Mister
other
Mister
observe
MistralAI-Index
ai crawler
Mistral
MistralAI-Index
observe