Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
NTENTbot
other
NTENTbot
observe
Nuzzel
other
Nuzzel
observe
officestorebot
other
officestorebot
observe
oii-research
other
oii-research
observe
Omgilibot
other
Omgilibot
observe
Omnisend
other
Omnisend
Omnisend
allow
Openbot
other
Openbot
observe
openfind
other
openfind
observe
Openfind data gatherer
other
observe
OpenGraphCheck
other
OpenGraphCheck
observe
OpenHoseBot
other
OpenHoseBot
observe
OpenTheBoxBot
other
OpenTheBoxBot
observe
opinion-tracker
other
opinion-tracker
observe
Oracle Ultra Search
other
Oracle
observe
OrangeBot
other
OrangeBot
observe
Orbbot
other
Orbbot
observe
Orthogaffe
other
Orthogaffe
observe
OutclicksBot
other
OutclicksBot
observe
OWLer-W
other
OWLer-W
observe
Paddle Webhooks
other
Paddle
Paddle
allow
Page2RSS
other
Page2RSS
observe
PageThing
other
PageThing
observe
Paqlebot
other
Paqle
Paqlebot
observe
parisbot
other
parisbot
observe
PayPal
other
PayPal
PayPal
allow
PDFBot
other
PDFBot
observe
peer39_crawler
other
Peer39
peer39_crawler
observe
PerMan
other
PerMan
observe
PiBot
other
PiBot
observe
PiplBot
other
PiplBot
observe
Pixalate.com
other
Pixalate.com
observe
PlagAwareBot
other
PlagAwareBot
observe
Plesk screenshot bot
other
Plesk
observe
Poduptime
other
Poduptime
observe
Polar
other
Polar
allow
Pomothy-Bot
other
Pomothy-Bot
observe
Poseidon Research Crawler
other
observe
postrank
other
postrank
observe
PR-CY.RU
other
PR-CY.RU
observe
PriEcoBot
other
PriEcoBot
observe
Primalbot
other
Primalbot
observe
PrimeHealthBot
other
PrimeHealthBot
observe
PrintFriendly
other
PrintFriendly.com
PrintFriendly.com
allow
PrivacyAwareBot
other
PrivacyAwareBot
observe
Pro Sitemaps
other
PRO Sitemaps
Pro
allow
PromptingBot
other
PromptingBot
observe
ProPowerBot
other
ProPowerBot
observe
ProWebWalker
other
ProWebWalker
observe
proxem
other
proxem
observe
psbot
other
psbot
observe
PubMatic Crawler Bot
other
PubMatic
observe
PulsePoint
other
PulsePoint
observe
PulsePoint-Crawler
other
PulsePoint-Crawler
observe
purebot
other
purebot
observe
PWABuilder
other
Microsoft
PWABuilderHttpAgent
observe
QueryN Metasearch
other
QueryN
observe
QuickPay Callback
other
QuickPay
allow
Qwam content intelligence
other
Qwam
observe
Radiation Retriever 1.1
other
Radiation
observe
Rakuten Image extraction bot
other
Rakuten
rakutenusabot-image
observe