Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
Goodreads
Goodreads
Google-adstxt
Google-Agent
GoogleAgent-URLContext
GoogleDocs
GoogleMessages
GoogleProducer
Mega Impacts
GovAPIEnricher
Grammarly
Grammarly
Grapeshot
GrapeshotCrawler
Groovinads
GroovinaAdsbot
GrowthX
GrowthXBot
CLERK SAS
Haiku-SearchBot
Hank
Hatena
Hatena::RUssia::Crawler
Hatena
HatenaBlog-bot
Check Point Software Technologies Ltd.
hecbot
Henry Labs
henry-shopping-agent
HubSpot
HubSpot-FeedFetcher
Integral Ad Science
ias-ie
IBM
IBM-Watson-NLU
SpazioDati
IDG/IT
IFTTT
IFTTT
Hive
ImagesiftBot
Apple
iMessage
Reticle
IncumbINTBot
Inoreader
Inoreader
Innologica
inoreader.com
Instapaper
Instaparser
IntentBeamBot
OneTag
InteractiveAdvertisingBureau
Subspace
InterviewOS
IntricleBot
IONOS
IonCrawl
JusProg
Jugendschutzprogramm-Crawler
Kagi
kagi-fetcher
Kargo
KargoBot
Kargo
KargoBot-Artemis-Mobile
Kernel
kernel
KIME
KimeBot
Moonshot AI
Kimi-User
Known Agents
Known
Kontur
Konturbot
URL Classification
KStandBot
LeadFinderBot
NewSiteLead
LeadForge
Lighthouse
LighthouseBot
Lightspeed Systems
LightspeedSystemsCrawler
Liner
LinerBot
Link
link-cli
Meltwater
linkfluence
Interactive32
LinkPreview
Fortra
LinkWalker
ListSignal
ListSignalBot
LMArenaUnfurlBot
Magnet
Magnet.me-web
Brandwatch
magpie-crawler