Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
hgfAlphaXCrawl
HIFI
HIFIBot
Honeybadger
Honeybadger
Honeybadger Industries
HoneybadgerBot
HostTracker
HostTracker
Hotjar
Hotjar
Chromium
hstspreload-bot
HTTrack
HubSeedsBot
HubSpot
HubSpot
HubSpot
HubSpot Connect 2.0 (http:
HubSpot
HubSpot Feed Fetcher
HubSpot
HubSpot Page Fetcher
HubSpot
HubSpot-FeedFetcher
Huckabuy
Huckabot
humanlinks
Hype Machine
Hype
HypeStat
hypestat
HyScore
hyscore.io
Internet Archive
ia_archiver
Internet Archive
ia_archiver-web.archive.org
Wikimedia Foundation
IABot
Integral Ad Science
IAS
Integral Ad Science
ias-ie
ias-va
iAsk
iAskBot
Ai Search
iaskspider
Babbar
IbouBot
ICBot
NICT
NICT
ICC-Crawler
ichiro
Idealo internet GmbH
idealo-bot
SpazioDati
IDG/IT
Iframely
Iframely
IFTTT
IFTTT
IFTTT RSS Feed Servuce
IFTTT
Hive
ImagesiftBot
ByteDance
imageSpider
Apple
iMessage
IMJ-CompanyPage-Scraper
Impact.com
imrbot
IndeedBot
Indeed
IndeedJobBot
INETDEX-Bot
InfoNaviRobot
infoobot
infoseek
Infrawatch
Infrawatch
Innguma
Innguma
Innguma
Innguma Fetcher
Innologica
inoreader.com
Inoreader
Inoreader
Innologica
inoreader.com
Instant Paper
Instapaper
Instapaper
Instaparser