Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
198 results
Ezoic Inc
EzoicBot
Fedica
Fedicabot
Fed Reporter, Inc.
FedReporterDataBot
AdsBot-Google
Mediapartners-Google
Google Digital Asset Links
Google-Firebase
Storebot-Google
Google-AdWords-Express
Google-Adwords-Instant
Google-Display-Ads-Bot
Google-Structured-Data-Testing
Google-TravelAds-Live
Grapeshot
GrapeshotCrawler
Groovinads
GroovinaAdsbot
Hellowork
HelloworkJobPostingBot
HTTrack
Huckabuy
Huckabot
HypeStat
hypestat
Integral Ad Science
IAS
Idealo internet GmbH
idealo-bot
Indeed
IndeedJobBot
cognitiveSEO
James
Klaviyo
Klaviyo/1.0
Krawler
KrawlerBot
DoubleVerify
Leikibot
LibreCrawl
LINE
line-poker
Siteimprove
LinkCheck
Siteimprove
LinkChecker
linkchecker.pro
LinkChecker
linkchecker.pro
LinkChecker.pro
LinkCheckerBot
Linkdex
linkdexbot
Magnet.me
MagnetmeBot
marketgoo
MarketGoo
SEOptimer
MBCrawler
MediaMonitoringBot
MediaMonitoringBot
Determ
Determ
Mediavine
Mediavine Medatada Parser
MegaIndex
MegaIndex.ru
Meta
meta-externalads
MetaInspector
METAJob
MetaJobBot
MGID
MgidBot
MirrorWeb Ltd
MirrorWebCrawler
Missinglettr
MissinglettrBot
Majestic
MJ12bot
ModernizeYourWebsite
ModernizeBot
Motominer
MotoMinerBot
Moz
DotBot
Moz
rogerbot
newspaper
Open-source
NitroPack Ltd
Nitro-
Noor Digital Agency AB
noorobot
Nosto
NostoCrawlerBot
Nostra
nostra
Nutch