Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Googlebot-News
Googlebot-Video
GoogleDocs
GoogleMessages
GoogleOther
GoogleProducer
GoParserBot
GoPay.cz
GoPay
Mega Impacts
GovAPIEnricher
Gowikibot
OpenAI
GPTBot
Grafana Labs
grafana
Grafana Labs
synthetic-monitoring-agent
Grafana Labs
Grafana Labs
Grafana&
Grammarly
Grammarly
Grapeshot
GrapeshotCrawler
Grapeshot
GrapeshotCrawler
GraphEmbedScraper
Greppr
Greppr
Grobbot
xAI
xAI
grok-agent
xAI
GrokBot
Groovinads
GroovinaAdsbot
Groovinads
GroovinaAdsbot
GroupHigh
GrowthX
GrowthXBot
grub-client
grub.org
gsa-crawler
gslfbot
GTmetrix
GTmetrix
Guest Posts
GuestpostsBot
Gwene
CLERK SAS
Haiku-SearchBot
Haloscan
HaloBot
HanaleiBot
Hank
Qihoo 360
HaosouSpider
Hardenize
Hardenize
Hark
Harvest
Hatena
Hatena Antenna
Hatena
Hatena::RUssia::Crawler
Hatena
HatenaBlog-bot
HawaiiBot
HeadlessChrome
HeadOnlyScraper
HeadOnlyScraperV2
Check Point Software Technologies Ltd.
hecbot
HelloWork Group
HelloWork
Hellowork
HelloworkJobPostingBot
Valyu
HenkBot
Henry Labs
henry-shopping-agent
Internet Archive
heritrix
HetrixTools Inc
HetrixTools
37signals
hgfAlphaXCrawl