Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
GoogleDocs
ai assistant
GoogleDocs
observe
GoogleMessages
ai assistant
GoogleMessages
observe
rDNS
GoogleOther
ai crawler
GoogleOther
observe
rDNS
GoogleProducer
ai assistant
GoogleProducer
observe
rDNS
GoParserBot
other
GoParserBot
observe
Gowikibot
other
Gowikibot
observe
GPTBot
ai crawler
OpenAI
GPTBot
observe
grafana
ai coding agent
Grafana Labs
grafana
observe
Grammarly
ai assistant
Grammarly
Grammarly
observe
GrapeshotCrawler
ai assistant
Grapeshot
GrapeshotCrawler
observe
Grobbot
other
Grobbot
observe
GroovinaAdsbot
ai assistant
Groovinads
GroovinaAdsbot
observe
GroupHigh
other
GroupHigh
observe
grub-client
other
grub-client
observe
grub.org
other
grub.org
observe
gsa-crawler
other
gsa-crawler
observe
gslfbot
other
gslfbot
observe
Gwene
other
Gwene
observe
Haiku-SearchBot
ai assistant
CLERK SAS
Haiku-SearchBot
observe
signed
HaloBot
other
Haloscan
HaloBot
observe
HanaleiBot
other
HanaleiBot
observe
Harvest
other
Harvest
observe
Hatena::RUssia::Crawler
ai assistant
Hatena
Hatena::RUssia::Crawler
observe
HatenaBlog-bot
ai assistant
Hatena
HatenaBlog-bot
observe
HawaiiBot
other
HawaiiBot
observe
HeadlessChrome
other
HeadlessChrome
observe
HeadOnlyScraper
other
HeadOnlyScraper
observe
HeadOnlyScraperV2
other
HeadOnlyScraperV2
observe
hecBot
ai assistant
Check Point Software Technologies Ltd.
hecbot
observe
signed
HelloWork
other
HelloWork Group
HelloWork
observe
HenkBot
ai data provider
Valyu
HenkBot
observe
Henry Shopping Agent
ai assistant
Henry Labs
henry-shopping-agent
observe
signed
HetrixTools Uptime Monitoring Bot
other
observe
hgfAlphaXCrawl
other
hgfAlphaXCrawl
observe
HIFIBot
other
HIFI
HIFIBot
observe
Honeybadger Uptime Check
ai coding agent
Honeybadger
Honeybadger
observe
HSTS preload bot
other
Chromium
hstspreload-bot
observe
HTTP Banner Detection
other
observe
HubSeedsBot
other
HubSeedsBot
observe
HubSpot
ai assistant
HubSpot
HubSpot
observe
HubSpot Crawler
other
HubSpot
HubSpot Connect 2.0 (http:
observe
HubSpot-FeedFetcher
ai assistant
HubSpot
HubSpot-FeedFetcher
observe
humanlinks
other
humanlinks
observe
HyScore
other
HyScore
observe
hyscore.io
other
hyscore.io
observe
ias-ie
ai assistant
Integral Ad Science
ias-ie
observe
ias-va
other
ias-va
observe
iAskBot
ai agent
iAsk
iAskBot
observe
iaskspider
ai agent
Ai Search
iaskspider
observe
ICBot
other
ICBot
observe
ICC Crawler
ai crawler
NICT
observe
ICC-Crawler
ai scraper
NICT
ICC-Crawler
observe
ichiro
other
ichiro
observe
IDG/IT
ai assistant
SpazioDati
IDG/IT
observe
IFTTT
ai assistant
IFTTT
IFTTT
observe
ImagesiftBot
ai assistant
Hive
ImagesiftBot
observe
imageSpider
ai scraper
ByteDance
imageSpider
observe
iMessage
ai assistant
Apple
iMessage
observe
IMJ-CompanyPage-Scraper
other
IMJ-CompanyPage-Scraper
observe
Impact.com Agent
other
Impact.com
observe