Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
268 results · AI Assistant
Kargo
KargoBot
Kargo
KargoBot-Artemis-Mobile
Kernel
Moonshot AI
Kimi-User
Known Agents
Known
URL Classification
KStandBot
Lighthouse
LighthouseBot
Lightspeed Systems
LightspeedSystemsCrawler
Liner
LinerBot
Link
Meltwater
linkfluence
Interactive32
LinkPreview
Fortra
LinkWalker
LMArenaUnfurlBot
Magnet
Magnet.me-web
Brandwatch
magpie-crawler
SourceForge
MagpieRSS
Mailchimp
MailChimp
Mantis
Mantisbot
Manus
Mastodon
Mastodon
MContextual
McontextualBot
Mediapartners-Google
Mediavine
Mediavine
Mediavine
MediavineMetadataParser
Medium
Mediumbot-MetaTagFetcher
Meta
meta-externalfetcher
Microsoft
Microsoft
Miniflux
Miniflux
Mistral AI
MistralAI-User
MixRank
MixrankBot
Muck Rack
MuckRackFeedParser
Nano Interactive
NanoInteractive
Nava Labs
Nekuda
Netcraft
NetcraftSurveyAgent
Neticle
Neticle
NetSeer
NetSeer
Netvibes
Netvibes
Neuwo
NeuwoBot
Newslit
Newslitbot
Marfeel
NewsRoom.BI
Nextcloud
NextCloud
VK
OdklBot
Opengraph
OpenGraph.io
Outbound Buddy
outboundbuddy-enrichment
Owler
Owler
SalesViewer
page-preview-tool
PagePeeker
PagePeeker
Domainsbot
Pandalytics
Panscient
panscient.com
PartnerOptimizer
PartnerOptimizer-bot
Payhawk
Perplexity
Perplexity-User
Phind
PhindBot
Piqo
Piqo
Pocket Casts
Pocket
PocketParser
Podchaser
PodchaserParser
Podimo
Podimo