Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
grub.org
other
grub.org
observe
gsa-crawler
other
gsa-crawler
observe
gslfbot
other
gslfbot
observe
GTmetrix
monitoring
GTmetrix
GTmetrix
allow
GuestpostsBot
monitoring
Guest Posts
GuestpostsBot
allow
Gwene
other
Gwene
observe
Haiku-SearchBot
ai assistant
CLERK SAS
observe
signed
HaloBot
other
Haloscan
HaloBot
observe
HanaleiBot
other
HanaleiBot
observe
HaosouSpider
search engine
Qihoo 360
HaosouSpider
allow
Hardenize
security
Hardenize
Hardenize
allow
Harvest
other
Harvest
observe
Hatena
social media
Hatena
Hatena
allow
Hatena::RUssia::Crawler
ai assistant
Hatena
Hatena::RUssia::Crawler
observe
HatenaBlog-bot
ai assistant
Hatena
HatenaBlog-bot
observe
HawaiiBot
other
HawaiiBot
observe
HeadlessChrome
other
HeadlessChrome
observe
HeadOnlyScraper
other
HeadOnlyScraper
observe
HeadOnlyScraperV2
other
HeadOnlyScraperV2
observe
hecBot
ai assistant
Check Point Software Technologies Ltd.
observe
signed
HelloWork
other
HelloWork Group
HelloWork
observe
HelloworkJobPostingBot
seo crawler
Hellowork
HelloworkJobPostingBot
challenge
HenkBot
ai data provider
Valyu
HenkBot
observe
Henry Shopping Agent
ai assistant
Henry Labs
observe
signed
heritrix
archiver
Internet Archive
heritrix
allow
HetrixTools
monitoring
HetrixTools Inc
HetrixTools
allow
HEY Email Privacy Proxy
social media
37signals
allow
hgfAlphaXCrawl
other
hgfAlphaXCrawl
observe
HIFIBot
other
HIFI
HIFIBot
observe
Honeybadger
ai coding agent
Honeybadger
Honeybadger
observe
honeybadgerbot
monitoring
Honeybadger Industries
HoneybadgerBot
allow
HostTracker
monitoring
HostTracker
HostTracker
allow
Hotjar
monitoring
Hotjar
Hotjar
allow
HSTS preload bot
other
Chromium
hstspreload-bot
observe
HTTP
other
HTTP
observe
HTTrack
seo crawler
HTTrack
challenge
HubSeedsBot
other
HubSeedsBot
observe
HubSpot Crawler
other
HubSpot
HubSpot
observe
HubSpot Feed Fetcher
social media
HubSpot
HubSpot
allow
HubSpot Page Fetcher
social media
HubSpot
HubSpot
allow
HubSpot-FeedFetcher
ai assistant
HubSpot
HubSpot-FeedFetcher
observe
Huckabuy Bot
seo crawler
Huckabuy
Huckabot
challenge
humanlinks
other
humanlinks
observe
Hype
search engine
Hype Machine
Hype
allow
Hype Machine
search engine
Hype Machine
allow
hypestat
seo crawler
HypeStat
hypestat
challenge
HyScore
other
HyScore
observe
hyscore.io
other
hyscore.io
observe
ia_archiver
archiver
Internet Archive
ia_archiver
allow
ia_archiver-web.archive.org
archiver
Internet Archive
ia_archiver-web.archive.org
allow
IABot
archiver
Wikimedia Foundation
IABot
allow
IAS crawler
seo crawler
Integral Ad Science
IAS
challenge
ias-ie
ai assistant
Integral Ad Science
ias-ie
observe
ias-va
other
ias-va
observe
iAskBot
ai agent
iAsk
iAskBot
observe
iaskspider
ai agent
Ai Search
iaskspider
observe
IbouBot
search engine
Babbar
IbouBot
allow
ICBot
other
ICBot
observe
ICC Crawler
ai crawler
NICT
observe
ICC-Crawler
ai scraper
NICT
ICC-Crawler
observe