Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
LighthouseBot
ai assistant
Lighthouse
LighthouseBot
observe
LightspeedSystemsCrawler
ai assistant
Lightspeed Systems
LightspeedSystemsCrawler
observe
LINE OGP Scraper
seo crawler
LINE
line-poker
challenge
LINER Bot
ai crawler
Liner Bot
LinerBot
observe
LinerBot
ai assistant
Liner
LinerBot
observe
Linespider
search engine
LINE Corporation
Linespider
allow
Linguee Bot
other
Linguee
observe
Link CLI
ai assistant
Link
link-cli
observe
signed
Link Tiger
monitoring
LinkTiger, Inc.
LinkTiger
allow
linkapediabot
other
linkapediabot
observe
LinkArchiver
other
LinkArchiver
observe
LinkCheck
seo crawler
Siteimprove
LinkCheck
challenge
LinkCheck by Siteimprove
seo crawler
Siteimprove
challenge
LinkCheck by Siteimprove.com
other
observe
LinkChecker Bot
seo crawler
LinkChecker
linkchecker.pro
challenge
linkchecker.pro
seo crawler
LinkChecker
linkchecker.pro
challenge
LinkCheckerBot
seo crawler
LinkChecker.pro
LinkCheckerBot
challenge
linkdex
other
linkdex
observe
linkdexbot
seo crawler
Linkdex
linkdexbot
challenge
LinkedInBot
social media
LinkedInBot
allow
LinkextractorPro
other
LinkextractorPro
observe
linkfluence
ai assistant
Meltwater
linkfluence
observe
Linkhub-crawler
other
Linkhub-crawler
observe
LinkisBot
other
LinkisBot
observe
linko
other
linko
observe
LinkpadBot
other
LinkpadBot
observe
LinkPreview
ai assistant
Interactive32
LinkPreview
observe
LinkRing
other
LinkRing
observe
LinkScan
other
LinkScan
observe
LinksIndexerBot
social media
Links Indexer
LinksIndexerBot
allow
LinkupBot
ai crawler
Linkup
LinkupBot
observe
LinkWalker
ai assistant
Fortra
LinkWalker
observe
lipperhey
other
lipperhey
observe
ListSignalBot
ai assistant
ListSignal
ListSignalBot
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe
LLMS-Txt-Scanner
other
LLMS-Txt-Scanner
observe
LMArenaUnfurlBot
ai assistant
LMArenaUnfurlBot
observe
LNSpiderguy
other
LNSpiderguy
observe
LocalDomainCrawler
other
LocalDomainCrawler
observe
logicmonitor
monitoring
LogicMonitor
LogicMonitor InternalService
allow
LogicMonitor SiteMonitor
ai coding agent
LogicMonitor
observe
LohiSoftBot
search engine
LohiSoft
LohiSoftBot
allow
Longtail News RSS Reader Bot
other
Longtail
observe
Loomly Bot
social media
Loomly
LoomlyBot
allow
lssbot
other
lssbot
observe
lssrocketcrawler
other
lssrocketcrawler
observe
ltx71
other
ltx71
observe
LumiaBot
other
LumiaBot
observe
Luminator-robots
other
Luminator-robots
observe
Luxxlebot
search engine
Luxxle
Luxxlebot
allow
lwp-trivial
other
lwp-trivial
observe
lyonl
search engine
Lyonl
lyonl
allow
lyonl-crawler
search engine
Lyonl
lyonl-crawler
allow
MaCoCu
other
MaCoCu
observe
Macrobondbot
monitoring
Macrobondbot
Macrobondbot
allow
MagiBot
search engine
Peak Labs
MagiBot
allow
Magnet.me-web
ai assistant
Magnet
Magnet.me-web
observe
MagnetmeBot
seo crawler
Magnet.me
MagnetmeBot
challenge
magpie-crawler
ai assistant
Brandwatch
magpie-crawler
observe