Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
psbot
other
psbot
observe
PubMatic
other
PubMatic
observe
PulsePoint
other
PulsePoint
observe
PulsePoint-Crawler
other
PulsePoint-Crawler
observe
Pulsetic
monitoring
Pulsetic
pulsetic
allow
pulsetic.com
ai coding agent
Pulsetic
pulsetic.com
observe
purebot
other
purebot
observe
PWABuilder
other
Microsoft
observe
PWABuilderHttpAgent
ai coding agent
Microsoft
PWABuilderHttpAgent
observe
QAtechBot
ai assistant
QA.tech
observe
qcbot
ai assistant
QUIC.cloud
qcbot
observe
QualifiedBot
ai crawler
Qualified.com, Inc.
QualifiedBot
observe
Qualys
security
Qualys
Qualys
allow
Qualys SSL Scanner
monitoring
Qualys
Qualys
allow
Quantcastbot
seo crawler
Quantcast
Quantcastbot
challenge
Quartr Crawler
seo crawler
Quartr
challenge
Querit-SearchBot
ai data provider
Querit
Querit-SearchBot
observe
QueritBot
ai data provider
Querit
QueritBot
observe
QueryN
other
QueryN
observe
QuickPay Callback
other
QuickPay
allow
Quora-Bot
ai assistant
Quora
Quora-Bot
observe
QuuuBot
ai assistant
Quuu
QuuuBot
observe
Qwam
other
Qwam
observe
Qwantbot
search engine
Qwant
Qwantbot
allow
Qwantbot-official
search engine
Qwant
Qwantbot-official
allow
Qwantify
search engine
Qwant
Qwantify
allow
Qwarrybot
ai assistant
Qwarry
Qwarrybot
observe
QwenBot
ai scraper
Alibaba
QwenBot
observe
Rackspace
monitoring
Rackspace
Rackspace
allow
Radiation
other
Radiation
observe
Rakuten
other
Rakuten
Rakuten
observe
Rakuten Image extraction bot
other
Rakuten
rakutenusabot-image
observe
RankActiveLinkBot
other
RankActiveLinkBot
observe
RankFlex
other
RankFlex
observe
RankMath
seo crawler
Rank Math
RankMath
challenge
RankurBot
ai assistant
Rankur
RankurBot
observe
RasaBot
other
RasaBot
observe
RavenCrawler
seo crawler
Raven
RavenCrawler
challenge
rawweb-bot
other
rawweb-bot
observe
Razorpay Webhook
other
Razorpay
allow
Readable
other
Added Bytes Ltd
Readable
allow
Readwise
ai assistant
Readwise
Readwise
observe
Readybot.io
ai assistant
Readybot
Readybot.io
observe
RecordedFuture
other
RecordedFuture
observe
Recurly Webhooks
other
Recurly
allow
Redactus Proxy Service
ai assistant
Redactus
observe
signed
Redactus Web Classifier
security
Redactus
allow
signed
redditbot
ai assistant
redditbot
observe
Reelevant
seo crawler
Reelevant
Reelevant
challenge
Refindbot
other
Refindbot
observe
RegionStuttgartBot
other
RegionStuttgartBot
observe
RepoMonkey
other
RepoMonkey
observe
research-bot
other
research-bot
observe
research-crawler
other
research-crawler
observe
research/recap
other
research/recap
observe
Retool
other
Retool
Retool
observe
RetrevoPageAnalyzer
other
RetrevoPageAnalyzer
observe
RetroListeCOM
monitoring
Niclas Papst
RetroListeCOM
allow
ReverseEngineeringBot
other
ReverseEngineeringBot
observe
Revvim
seo crawler
Revvim
challenge