Skip to main content
Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

497 results

NewsNow search engine

NewsNowUK

NewsNow
allow
Nicecrawler archiver

NiceCrawler

Nicecrawler
allow
NixStatsMonitoringBot monitoring

NixStats

NixStatsMonitoringBot
allow
Nmap Scripting Engine security

Nmap

Nmap
allow
Nodeping monitoring

Nodeping

NodePing
allow
Noibu JS Beautifier other

Noibu

Noibu
allow
Nooshub social media

Nooshub

Nooshub
allow
Notabot search engine

Helpfeel

Notabot
allow
Notion Automation other

Notion

NotionAutomation
allow
Observer security

Observer

Observer
allow
OddlyBot archiver

Oddly Even

OddlyBot
allow
Odin security

ODIN

Odin
allow
Oh Dear monitoring

Oh Dear

OhDear
allow
OKX-dolphin-crawler security

OKX

OKX-dolphin-crawler
allow
Omnisend other

Omnisend

Omnisend
allow
OneTrust security

OneTrust

OneTrust
allow
Onetrust CMP Scanner monitoring

OneTrust, LLC.

Onetrust
allow
OnlineOrNot Monitor monitoring

OnlineOrNot

OnlineOrNot.com_bot_1.0_(https:
allow
OpenBash-Surface security

OpenBash

OpenBash-Surface
allow
Opengraph Bot social media

Opengraph

allow
OpenRSS social media

OpenRSS

OpenRSS
allow
Orlo Link Preview social media

Orlo

Orlo-LinkPreview
allow
OsanoBot security

Osano

OsanoBot
allow
OurFamilyWizard social media

OurFamilyWizard

Cozi-iCalendar-FeedReader
allow
OutsellURLValidator monitoring

Outsell Corporation

OutsellURLValidator
allow
Overcast social media

Overcast Radio

Overcast
allow
Ozon Web Grabber social media

Ozon

Ozon
allow
Paddle Webhooks other

Paddle

Paddle
allow
PaesslerCloudBot monitoring

Paessler

PRTGCloudBot
allow
PageSourceSearchBot search engine

PageSourceSearch

PageSourceSearchBot
allow
Panopta monitoring

Panopta

Panopta
allow
Parse.ly monitoring

Parse.ly

parse.ly scraper
allow
PayPal other

PayPal

PayPal
allow
Pingdom monitoring

Pingdom

Pingdom
allow
PingPing monitoring

PingPing

pingping.io/
allow
Pocket Casts Feed Parser social media

Pocket Casts

Pocket
allow
Polar other

Polar

allow
Potions search engine

Potions

Potions
allow
PressEngine Bot social media

PressEngine

PressEngine Bot
allow
PrintFriendly other

PrintFriendly.com

PrintFriendly.com
allow
Pro Sitemaps other

PRO Sitemaps

Pro
allow
Probely security

Probely

ProbelySPDR
allow
Project Honeypot security

Unspam Technologies, Inc

Verispider
allow
ProjectShield Url Check security

Google

ProjectShield-UrlCheck
allow
Protopage social media

Protopage Ltd

Protopage
allow
PublicWWWBot search engine

PublicWWW

PublicWWWBot
allow
Pulsetic monitoring

Pulsetic

pulsetic.com (+https:
allow
PurityBot search engine

Purili

PurityBot
allow
Qualys security

Qualys

cf-qualys-scanner
allow
Qualys SSL Scanner monitoring

Qualys

Qualys SSL Scanner
allow
QuickPay Callback other

QuickPay

allow
Qwantbot search engine

Qwant

Qwantbot
allow
Qwantbot-official search engine

Qwant

Qwantbot-official
allow
Qwantify search engine

Qwant

Qwantify
allow
Rackspace monitoring

Rackspace

Rackspace Monitoring
allow
Razorpay Webhook other

Razorpay

Razorpay
allow
Readable other

Added Bytes Ltd

Readable
allow
Recurly Webhooks other

Recurly

allow
Redactus Web Classifier security

Redactus

RedactusWebClassifier
allow signed
RetroListeCOM monitoring

Niclas Papst

RetroListeCOM
allow
← Prev 6 / 9 Next →