Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
HIFI
HIFIBot
Honeybadger
Honeybadger
Honeybadger Industries
HoneybadgerBot
HostTracker
HostTracker
Hotjar
Hotjar
Chromium
hstspreload-bot
HTTrack
HubSeedsBot
HubSpot
HubSpot Connect 2.0 (http:
HubSpot
HubSpot Feed Fetcher
HubSpot
HubSpot Page Fetcher
HubSpot
HubSpot-FeedFetcher
Huckabuy
Huckabot
humanlinks
HUNT-Bot
Tencent
Hunyuan
Hype Machine
Hype
HypeStat
hypestat
HyScore
hyscore.io
Internet Archive
ia_archiver
Internet Archive
ia_archiver-web.archive.org
Wikimedia Foundation
IABot
Integral Ad Science
IAS
Integral Ad Science
ias-ie
ias-va
iAsk
iAskBot
Ai Search
iaskspider
IBM
IBM-Watson-NLU
Ibou
ibou-discovery
Ibou
ibou-images
Ibou
ibou-news
Ibou
ibou-search
Babbar
IbouBot
ICBot
NICT
NICT
ICC-Crawler
ichiro
Idealo internet GmbH
idealo-bot
SpazioDati
IDG/IT
Iframely
Iframely
IFTTT
IFTTT
IFTTT RSS Feed Servuce
IFTTT
ImageBot
Hive
ImagesiftBot
ByteDance
imageSpider
Apple
iMessage
IMJ-CompanyPage-Scraper
Impact.com
imrbot
Reticle
IncumbINTBot
IndeedBot
Indeed
IndeedJobBot
INETDEX-Bot
InfoNaviRobot
infoobot
infoseek
Infrawatch
Infrawatch