Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Google-Structured-Data-Testing-Tool
Google-TravelAds-Live
google-xrawler
Google DeepMind
GoogleAgent-Mariner
GoogleAgent-URLContext
GoogleAssociationService
Googlebot
Googlebot-IA
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
GoogleDocs
GoogleMessages
GoogleOther
GoogleProducer
GoParserBot
GoPay.cz
GoPay
Gowikibot
OpenAI
GPTBot
Grafana Labs
grafana
Grafana Labs
synthetic-monitoring-agent
Grafana Labs
Grammarly
Grammarly
Grapeshot
GrapeshotCrawler
Grapeshot
GrapeshotCrawler
Greppr
Greppr
Grobbot
Groovinads
GroovinaAdsbot
Groovinads
GroovinaAdsbot
GroupHigh
grub-client
grub.org
gsa-crawler
gslfbot
GTmetrix
GTmetrix
Guest Posts
GuestpostsBot
Gwene
CLERK SAS
Haiku-SearchBot
Haloscan
HaloBot
HanaleiBot
Qihoo 360
HaosouSpider
Hardenize
Hardenize
Harvest
Hatena
Hatena Antenna
Hatena
Hatena::RUssia::Crawler
Hatena
HatenaBlog-bot
HawaiiBot
HeadlessChrome
HeadOnlyScraper
HeadOnlyScraperV2
Check Point Software Technologies Ltd.
hecbot
HelloWork Group
HelloWork
Hellowork
HelloworkJobPostingBot
Valyu
HenkBot
Henry Labs
henry-shopping-agent
Internet Archive
heritrix
HetrixTools Inc
HetrixTools
37signals