Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
Qwairy
Qwairy-Bot
Qwarry
Qwarrybot
Rankshift
RankshiftFetcher
Rankur
RankurBot
Readwise
Readwise
Readybot
Readybot.io
Redactus
redactus-proxy-service
redditbot
Nuvo
revops-nuvo-detector
RSS.Social
RSS.Social
RSSExtractor
Rye
ryebot
Scalinth
ScalinthBot
Scavo
ScavoBot
Meltwater
scoop.it
Scoop.it
scoopit-crawler
Scope3
Scope3
Scour
scour.ing
Scour
ScourRSSBot
The New York Times
[email protected]
Semantic Visions
semantic-visions-analytica
Semantic Visions
semantic-visions-infoseeker
Semantic Visions
SemanticVisionsBot
Sender Online Marketing
SenderBot
SentiOne
SentiBot
Serendeputy
serendeputybot
Salseforce
SFDC-Callout
Parallel
Shap-User
ShortPixel
ShortPixel
Silverstream AI
silvestro
SimplePie
SimplePie
Site Stats Database
SiteStatsBot
Slack
Slack-ImgProxy
Slack
Slackbot-LinkExpanding
Slickstream
SlickBot
Christopher Albert
slopsearch
SMBsBot
Similarweb
SMTBot
Snap
Snap
Snapchat
Snapchat
UpContent
Sottopop
UpContent
Sottopop-user-rss
UpContent
SottopopNone
SpamCipher
SpamCipherBot
Speroll Media
Speroll-AdsTxt-Crawler
SpyglassesBot
Start.me
StartmeBot
Bryan Norcross
StormIntelCrawler
Substack
SubstackContentFetch
Superfeedr
Superfeedr
Sur.ly
SurdotlyBot
Matrix
Synapse
Taboola
Taboolabot
TechSpy
TechSpyBot
TextRazor
TheSkinnyCrawler
TheWebReport
ByteDance
TikTokSpider
Tiny Tiny RSS
Tiny
Titan Materials Corp
TitanBot