Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
290 results · AI Assistant
NewsBlur
Newslit
Newslitbot
Marfeel
NewsRoom.BI
Nextcloud
NextCloud
VK
OdklBot
Opengraph
OpenGraph.io
Orlo
Orlo-LinkPreview
Outbound Buddy
outboundbuddy-enrichment
Owler
Owler
SalesViewer
page-preview-tool
PagePeeker
PagePeeker
Domainsbot
Pandalytics
Panscient
panscient.com
PartnerOptimizer
PartnerOptimizer-bot
Payhawk
payhawk-invoice-fetching-bot
Perplexity
Perplexity-User
Perplexity
Perplexity-User
Phind
PhindBot
Piqo
Piqo
PocketParser
Podchaser
PodchaserParser
Podimo
Podimo
Poggio
Poggio-Citations
PressEngine
PressEngineBot
Productsup
productsup.io/crawler
QA.tech
QATechBot
QUIC.cloud
qcbot
Quora
Quora-Bot
Quuu
QuuuBot
Qwarry
Qwarrybot
Rankur
RankurBot
Readwise
Readwise
Readybot
Readybot.io
Redactus
redactus-proxy-service
redditbot
RSS.Social
RSS.Social
Rye
ryebot
Scavo
ScavoBot
Meltwater
scoop.it
Scoop.it
scoopit-crawler
Scope3
Scope3
Scour
scour.ing
Scour
ScourRSSBot
The New York Times
[email protected]
SentiOne
SentiBot
Serendeputy
serendeputybot
Salseforce
SFDC-Callout
Parallel
Shap-User
ShortPixel
ShortPixel
Silverstream AI
silvestro
SimplePie
SimplePie
Slack
Slack-ImgProxy
Slack
Slackbot-LinkExpanding
Slack
SlackbotLinkExpanding
Slickstream
SlickBot
Christopher Albert
slopsearch
Similarweb
SMTBot
Snap
Snap
Snapchat
Snapchat
UpContent
Sottopop