Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
497 results
Google-PageRenderer
Google-BusinessLinkVerifi
Google-Flights-Search-Spackle
Google-NotebookLM
google-xrawler
Googlebot
Googlebot-IA
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
GoPay.cz
GoPay
Grafana Labs
synthetic-monitoring-agent
Grafana Labs
Greppr
Greppr
GTmetrix
GTmetrix
Guest Posts
GuestpostsBot
Qihoo 360
HaosouSpider
Hardenize
Hardenize
Hatena
Hatena Antenna
Internet Archive
heritrix
HetrixTools Inc
HetrixTools
37signals
Honeybadger Industries
HoneybadgerBot
HostTracker
HostTracker
Hotjar
Hotjar
HubSpot
HubSpot Feed Fetcher
HubSpot
HubSpot Page Fetcher
Hype Machine
Hype
Internet Archive
ia_archiver
Internet Archive
ia_archiver-web.archive.org
Wikimedia Foundation
IABot
Ibou
ibou-discovery
Ibou
ibou-images
Ibou
ibou-news
Ibou
ibou-search
Babbar
IbouBot
Iframely
Iframely
IFTTT RSS Feed Servuce
IFTTT
Infrawatch
Infrawatch
Innguma
Innguma Fetcher
Innologica
inoreader.com
Make
Integromat
Intelligence X
intelx.io_bot
Internet Archive
archive.org_bot
Archive-It
Archive-It
Internet Archive
InternetArchiveBot
Driftnet
InternetMeasurement
IsDownBot
IsDownBot
ISS
ISSCyberRiskCrawler
Jagged Pixel Inc.
www.getuptime.co
JamBot
Jambot
Automattic
Jetpack
Jobiglo
Jobiglo-Bot
JobsWithGPT
jobswithgptcom-bot
Jooble
Jooblebot
Coinbase
Jumio
K7 Computing
K7MLWCBot
Kagi
Kagi
Kagibot