Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
290 results · AI Assistant
Hatena
Hatena::RUssia::Crawler
Hatena
HatenaBlog-bot
Check Point Software Technologies Ltd.
hecbot
Henry Labs
henry-shopping-agent
HubSpot
HubSpot
HubSpot
HubSpot-FeedFetcher
Integral Ad Science
ias-ie
SpazioDati
IDG/IT
IFTTT
IFTTT
Hive
ImagesiftBot
Apple
iMessage
Innguma
Innguma
Inoreader
Inoreader
Innologica
inoreader.com
Instapaper
Instaparser
OneTag
InteractiveAdvertisingBureau
IONOS
IonCrawl
JusProg
Jugendschutzprogramm-Crawler
Kagi
kagi-fetcher
Kargo
KargoBot
Kargo
KargoBot-Artemis-Mobile
Kernel
kernel
Moonshot AI
Kimi-User
Known Agents
Known
URL Classification
KStandBot
Lighthouse
LighthouseBot
Lightspeed Systems
LightspeedSystemsCrawler
Liner
LinerBot
Link
link-cli
Meltwater
linkfluence
Interactive32
LinkPreview
Fortra
LinkWalker
LMArenaUnfurlBot
Magnet
Magnet.me-web
Brandwatch
magpie-crawler
SourceForge
MagpieRSS
Mailchimp
MailChimp
Mantis
Mantisbot
Manus
manus-bot
Mastodon
Mastodon
MContextual
McontextualBot
Mediapartners-Google
Mediavine
MediavineMetadataParser
Medium
Mediumbot-MetaTagFetcher
Meta
meta-externalfetcher
Miniflux
Miniflux
Mistral AI
MistralAI-User
MixRank
MixrankBot
Muck Rack
MuckRack
Muck Rack
MuckRackFeedParser
Nano Interactive
NanoInteractive
Nava Labs
Nekuda
nekuda-payment-executor
Netcraft
NetcraftSurveyAgent
netEstate
Neticle
Neticle
NetSeer
NetSeer
Netvibes
Netvibes
Neuwo
NeuwoBot
NewsBlur