Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
471 results
Grafana Labs
synthetic-monitoring-agent
Grafana Labs
Greppr
Greppr
GTmetrix
GTmetrix
Guest Posts
GuestpostsBot
Qihoo 360
HaosouSpider
Hardenize
Hardenize
Hatena
Hatena Antenna
Internet Archive
heritrix
HetrixTools Inc
HetrixTools
37signals
Honeybadger Industries
HoneybadgerBot
HostTracker
HostTracker
Hotjar
Hotjar
HubSpot
HubSpot Feed Fetcher
HubSpot
HubSpot Page Fetcher
Hype Machine
Hype
Internet Archive
ia_archiver
Internet Archive
ia_archiver-web.archive.org
Wikimedia Foundation
IABot
Babbar
IbouBot
Iframely
Iframely
IFTTT RSS Feed Servuce
IFTTT
Infrawatch
Infrawatch
Innguma
Innguma Fetcher
Innologica
inoreader.com
Make
Integromat
Intelligence X
intelx.io_bot
Internet Archive
archive.org_bot
Archive-It
Archive-It
Internet Archive
InternetArchiveBot
Driftnet
InternetMeasurement
IsDownBot
IsDownBot
ISS
ISSCyberRiskCrawler
Jagged Pixel Inc.
www.getuptime.co
JamBot
Jambot
Automattic
Jetpack
Jobiglo
Jobiglo-Bot
JobsWithGPT
jobswithgptcom-bot
Jooble
Jooblebot
Coinbase
Jumio
K7 Computing
K7MLWCBot
Kagi
Kagi
Kagibot
Kakao
kakaotalk-scrap
Kargo
KargoBot-Artemis
Kinsta
kinsta-bot
Legal Monster ApS
LegalMonster
Let's Encrypt
Let's Encrypt
Level 9
Level9SearchBot
LINE Corporation
Linespider
LinkTiger, Inc.
LinkTiger
LinkedInBot
Links Indexer
LinksIndexerBot
LogicMonitor
LogicMonitor InternalService
Loomly
LoomlyBot
Luxxle
Luxxlebot
Lyonl
lyonl
Lyonl
lyonl-crawler