Skip to main content
Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

471 results

OurFamilyWizard social media

OurFamilyWizard

Cozi-iCalendar-FeedReader
allow
OutsellURLValidator monitoring

Outsell Corporation

OutsellURLValidator
allow
Overcast social media

Overcast Radio

Overcast
allow
Ozon Web Grabber social media

Ozon

Ozon
allow
Paddle Webhooks other

Paddle

Paddle
allow
PaesslerCloudBot monitoring

Paessler

PRTGCloudBot
allow
Panopta monitoring

Panopta

Panopta
allow
Parse.ly monitoring

Parse.ly

parse.ly scraper
allow
PayPal other

PayPal

PayPal
allow
Pingdom monitoring

Pingdom

Pingdom
allow
PingPing monitoring

PingPing

pingping.io/
allow
Pocket Casts Feed Parser social media

Pocket Casts

Pocket
allow
Polar other

Polar

allow
Potions search engine

Potions

Potions
allow
PressEngine Bot social media

PressEngine

PressEngine Bot
allow
PrintFriendly other

PrintFriendly.com

PrintFriendly.com
allow
Pro Sitemaps other

PRO Sitemaps

Pro
allow
Probely security

Probely

ProbelySPDR
allow
Project Honeypot security

Unspam Technologies, Inc

Verispider
allow
ProjectShield Url Check security

Google

ProjectShield-UrlCheck
allow
ProjectShield-UrlCheck security

Google

ProjectShield-UrlCheck
allow
Protopage social media

Protopage Ltd

Protopage
allow
Pulsetic monitoring

Pulsetic

pulsetic.com (+https:
allow
Qualys security

Qualys

cf-qualys-scanner
allow
Qualys SSL Scanner monitoring

Qualys

Qualys SSL Scanner
allow
QuickPay Callback other

QuickPay

allow
Qwantbot search engine

Qwant

Qwantbot
allow
Qwantbot-official search engine

Qwant

Qwantbot-official
allow
Qwantify search engine

Qwant

Qwantify
allow
Rackspace monitoring

Rackspace

Rackspace Monitoring
allow
Razorpay Webhook other

Razorpay

Razorpay
allow
Readable other

Added Bytes Ltd

Readable
allow
Recurly Webhooks other

Recurly

allow
Redactus Web Classifier security

Redactus

RedactusWebClassifier
allow signed
RetroListeCOM monitoring

Niclas Papst

RetroListeCOM
allow
Reward Gateway security

Reward Gateway

reward-gateway
allow
Robinbot monitoring

Robin Education

Robinbot
allow signed
RSS API social media

RSS API (by Tibush GmbH)

RSSAPI
allow
rss2tg_bot social media

Yellow Rubber Duck Consulting

rss2tg
allow
Sansec Security Monitor security

Sansec Security Monitor

Sansec
allow
Scour RSS Bot social media

Scour

scour.ing
allow
Scrubby search engine

Scrub the Web

Scrubby
allow
SE Ranking Bot monitoring

SE Ranking

SE
allow
search.marginalia.nu search engine

Marginalia

search.marginalia.nu
allow
SecurityHeaders security

Security Headers

SecurityHeaders
allow
Seekport search engine

SISTRIX

Seekport
allow
SeekportBot search engine

SISTRIX

SeekportBot
allow
SemanticScholarBot search engine

Ai2

SemanticScholarBot
allow
SendGrid other

SendGrid

allow
Sentry security

Sentry

https://sentry.io
allow
Sentry Uptime Monitoring monitoring

Sentry Uptime Bot

Sentry
allow
SequelWP monitoring

SequelWP

SequelWP
allow
Server Density monitoring

Stackpath

Server
allow
Seznam search engine

Seznam

Android(10);cz.seznam.sbrowser;12.6.0
allow
SeznamBot search engine

Seznam

SeznamBot
allow
Shopify-Captain-Hook other

Shopify

Shopify-Captain-Hook
allow
Shortwave Image Fetcher social media

Shortwave Communications Inc.

Shortwave
allow
SilktideBot other

Silktide

SilktideBot
allow
SISTRIX Optimizer monitoring

SISTRIX GmbH

allow
Site24x7 monitoring

Site24x7

site24x7
allow
← Prev 6 / 8 Next →