Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
Primalbot
other
Primalbot
observe
PrimeHealthBot
other
PrimeHealthBot
observe
PrivacyAwareBot
other
PrivacyAwareBot
observe
productsup.io/crawler
ai assistant
Productsup
productsup.io/crawler
observe
PromptingBot
other
PromptingBot
observe
ProPowerBot
other
ProPowerBot
observe
ProWebWalker
other
ProWebWalker
observe
proxem
other
proxem
observe
PRTGCloudBot
ai coding agent
Paessler
PRTGCloudBot
observe
psbot
other
psbot
observe
PubMatic Crawler Bot
other
PubMatic
observe
PulsePoint
other
PulsePoint
observe
PulsePoint-Crawler
other
PulsePoint-Crawler
observe
pulsetic
ai coding agent
Pulsetic
pulsetic
observe
pulsetic.com
ai coding agent
Pulsetic
pulsetic.com
observe
purebot
other
purebot
observe
PWABuilder
other
Microsoft
PWABuilderHttpAgent
observe
PWABuilderHttpAgent
ai coding agent
Microsoft
PWABuilderHttpAgent
observe
QAtechBot
ai assistant
QA.tech
QATechBot
observe
qcbot
ai assistant
QUIC.cloud
qcbot
observe
QualifiedBot
ai crawler
Qualified.com, Inc.
QualifiedBot
observe
Querit-SearchBot
ai data provider
Querit
Querit-SearchBot
observe
QueritBot
ai data provider
Querit
QueritBot
observe
QueryN Metasearch
other
QueryN
observe
Quora-Bot
ai assistant
Quora
Quora-Bot
observe
QuuuBot
ai assistant
Quuu
QuuuBot
observe
Qwam content intelligence
other
Qwam
observe
Qwarrybot
ai assistant
Qwarry
Qwarrybot
observe
QwenBot
ai scraper
Alibaba
QwenBot
observe
Rackspace Monitoring
ai coding agent
Rackspace
observe
Radiation Retriever 1.1
other
Radiation
observe
Rakuten Image extraction bot
other
Rakuten
rakutenusabot-image
observe
RankActiveLinkBot
other
RankActiveLinkBot
observe
RankFlex
other
RankFlex
observe
RankurBot
ai assistant
Rankur
RankurBot
observe
RasaBot
other
RasaBot
observe
rawweb-bot
other
rawweb-bot
observe
Readwise
ai assistant
Readwise
Readwise
observe
Readybot.io
ai assistant
Readybot
Readybot.io
observe
RecordedFuture Global Inventory Crawler
other
RecordedFuture
observe
Redactus Proxy Service
ai assistant
Redactus
redactus-proxy-service
observe
signed
redditbot
ai assistant
redditbot
observe
rednote-websearch-bot
other
rednote-websearch-bot
observe
Refindbot
other
Refindbot
observe
Reflectionbot
ai scraper
Reflection
Reflectionbot
observe
RegionStuttgartBot
other
RegionStuttgartBot
observe
RepoMonkey
other
RepoMonkey
observe
RepoMonkey Bait & Tackle
other
observe
research-bot
other
research-bot
observe
research-crawler
other
research-crawler
observe
research/recap
other
research/recap
observe
ResearchBot
other
ResearchBot
observe
Retool
other
Retool
Retool
observe
RetrevoPageAnalyzer
other
RetrevoPageAnalyzer
observe
ReverseEngineeringBot
other
ReverseEngineeringBot
observe
RidderBot
other
RidderBot
observe
Riddler
other
Riddler
observe
Rivva
other
Rivva
observe
robots-ai-permissions
other
robots-ai-permissions
observe
Robozilla
other
Robozilla
observe