Updated daily
Crawler Catalog
1,948 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
825 results · Other
HaloBot
other
Haloscan
HaloBot
observe
HanaleiBot
other
HanaleiBot
observe
Harvest
other
Harvest
observe
HawaiiBot
other
HawaiiBot
observe
HeadlessChrome
other
HeadlessChrome
observe
HeadOnlyScraper
other
HeadOnlyScraper
observe
HeadOnlyScraperV2
other
HeadOnlyScraperV2
observe
HelloWork
other
HelloWork Group
HelloWork
observe
hgfAlphaXCrawl
other
hgfAlphaXCrawl
observe
HIFIBot
other
HIFI
HIFIBot
observe
HSTS preload bot
other
Chromium
hstspreload-bot
observe
HTTP
other
HTTP
observe
HubSeedsBot
other
HubSeedsBot
observe
HubSpot Crawler
other
HubSpot
HubSpot
observe
humanlinks
other
humanlinks
observe
HyScore
other
HyScore
observe
hyscore.io
other
hyscore.io
observe
ias-va
other
ias-va
observe
ICBot
other
ICBot
observe
ichiro
other
ichiro
observe
IMJ-CompanyPage-Scraper
other
IMJ-CompanyPage-Scraper
observe
Impact.com
other
Impact.com
observe
imrbot
other
imrbot
observe
IndeedBot
other
IndeedBot
observe
INETDEX-Bot
other
INETDEX-Bot
observe
InfoNaviRobot
other
InfoNaviRobot
observe
infoobot
other
infoobot
observe
infoseek
other
infoseek
observe
Instapaper
other
Instant Paper
Instapaper
observe
Integromat
other
Make
allow
integromedb
other
integromedb
observe
intelium_bot
other
intelium_bot
observe
InterfaxScanBot
other
InterfaxScanBot
observe
ip-web-crawler.com
other
ip-web-crawler.com
observe
ips-agent
other
ips-agent
observe
IRLbot
other
IRLbot
observe
Iron33
other
Iron33
observe
iskanie
other
iskanie
observe
IsraBot
other
IsraBot
observe
istellabot
other
istellabot
observe
it2media-domain-crawler
other
it2media-domain-crawler
observe
JamesBOT
other
JamesBOT
observe
Jamie's
other
Jamie's
observe
JenkersBot
other
JenkersBot
observe
JennyBot
other
JennyBot
observe
Jetbot
other
Jetbot
observe
Jetty
other
Jetty
observe
JikeSpider
other
JikeSpider
observe
JobboerseBot
other
JobboerseBot
observe
JobHuntBot
other
JobHuntBot
observe
jpg-newsbot
other
jpg-newsbot
observe
jyxobot
other
jyxobot
observe
k2spider
other
k2spider
observe
kaikki.org-digital-archive
other
kaikki.org-digital-archive
observe
KauaiBot
other
KauaiBot
observe
kbcrawl
other
kbcrawl
observe
Kemvibot
other
Kemvibot
observe
Kenjin
other
Kenjin
observe
Keydrop.io
other
Keydrop.io
observe
keys-so-bot
other
keys-so-bot
observe