Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
LinkextractorPro
other
LinkextractorPro
observe
linkfluence
ai assistant
Meltwater
linkfluence
observe
LinkisBot
other
LinkisBot
observe
linko
other
linko
observe
LinkpadBot
other
LinkpadBot
observe
LinkPreview
ai assistant
Interactive32
LinkPreview
observe
LinkRing
other
LinkRing
observe
LinkScan
other
LinkScan
observe
LinksIndexerBot
social media
Links Indexer
LinksIndexerBot
allow
LinkupBot
ai crawler
Linkup
LinkupBot
observe
LinkWalker
ai assistant
Fortra
LinkWalker
observe
lipperhey
other
lipperhey
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe
LMArenaUnfurlBot
ai assistant
LMArenaUnfurlBot
observe
LNSpiderguy
other
LNSpiderguy
observe
logicmonitor
monitoring
LogicMonitor
LogicMonitor InternalService
allow
LogicMonitor SiteMonitor
ai coding agent
LogicMonitor
observe
Longtail News RSS Reader Bot
other
Longtail
observe
Loomly Bot
social media
Loomly
LoomlyBot
allow
lssbot
other
lssbot
observe
lssrocketcrawler
other
lssrocketcrawler
observe
ltx71
other
ltx71
observe
Luminator-robots
other
Luminator-robots
observe
Luxxlebot
search engine
Luxxle
Luxxlebot
allow
lwp-trivial
other
lwp-trivial
observe
lyonl
search engine
Lyonl
lyonl
allow
lyonl-crawler
search engine
Lyonl
lyonl-crawler
allow
MaCoCu
other
MaCoCu
observe
Macrobondbot
monitoring
Macrobondbot
Macrobondbot
allow
MagiBot
search engine
Peak Labs
MagiBot
allow
Magnet.me-web
ai assistant
Magnet
Magnet.me-web
observe
MagnetmeBot
seo crawler
Magnet.me
MagnetmeBot
challenge
magpie-crawler
ai assistant
Brandwatch
magpie-crawler
observe
MagpieRSS
ai assistant
SourceForge
MagpieRSS
observe
Magus Bot
other
Magus
observe
MailChimp
ai assistant
Mailchimp
MailChimp
observe
MailRUBot
social media
Mail Russia
Mail.RU_Bot
allow
MainWP
other
Direct Support / Website Managed
MainWP
allow
Make Platform
other
Make
allow
Make.com
ai crawler
Make.com
make.com
observe
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
ManageWP
other
ManageWP
ManageWP
allow
Mantisbot
ai assistant
Mantis
Mantisbot
observe
Manus Bot
ai assistant
Manus
manus-bot
observe
Manus-User
browser automation
Butterfly Effect
Manus-User
observe
mappydata
other
mappydata
observe
Marginalia Search
search engine
Marginalia
search.marginalia.nu
allow
marketgoo
seo crawler
marketgoo
MarketGoo
challenge
Mars Finder
search engine
Mars Flag
Mars
allow
Mastodon
ai assistant
Mastodon
Mastodon
observe
MastodonInstances
other
MastodonInstances
observe
Mata Hari
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MauiBot
other
MauiBot
observe
Mavifinds
other
Brandimi
Mavifinds
observe
Mavifinds Bot
monitoring
Mavifinds
Mavifinds
allow
MBCrawler
seo crawler
SEOptimer
MBCrawler
challenge
McontextualBot
ai assistant
MContextual
McontextualBot
observe