Skip to main content
Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

497 results

Reward Gateway security

Reward Gateway

reward-gateway
allow
Robinbot monitoring

Robin Education

Robinbot
allow signed
RSS API social media

RSS API (by Tibush GmbH)

RSSAPI
allow
rss2tg_bot social media

Yellow Rubber Duck Consulting

rss2tg
allow
Sansec Security Monitor security

Sansec Security Monitor

Sansec
allow
Scour RSS Bot social media

Scour

scour.ing
allow
Scrubby search engine

Scrub the Web

Scrubby
allow
SE Ranking Bot monitoring

SE Ranking

SE
allow rDNS
search.marginalia.nu search engine

Marginalia

search.marginalia.nu
allow
SecurityHeaders security

Security Headers

SecurityHeaders
allow
Seekport search engine

SISTRIX

Seekport
allow
SeekportBot search engine

SISTRIX

SeekportBot
allow
SemanticScholarBot search engine

Ai2

SemanticScholarBot
allow
SendGrid other

SendGrid

allow
Sentry security

Sentry

https://sentry.io
allow
Sentry Uptime Monitoring monitoring

Sentry Uptime Bot

Sentry
allow
SequelWP monitoring

SequelWP

SequelWP
allow
Server Density monitoring

Stackpath

Server
allow
Seznam search engine

Seznam

Android(10);cz.seznam.sbrowser;12.6.0
allow
SeznamBot search engine

Seznam

SeznamBot
allow
Shopify-Captain-Hook other

Shopify

Shopify-Captain-Hook
allow
Shortwave Image Fetcher social media

Shortwave Communications Inc.

Shortwave
allow
SilktideBot other

Silktide

SilktideBot
allow
SISTRIX Optimizer monitoring

SISTRIX GmbH

allow
Site24x7 monitoring

Site24x7

site24x7
allow
SiteLock security

SiteLock

SiteLock (AccMan Connector)
allow
SiteLockSpider security

SiteLock

SiteLockSpider
allow
SiteSearch360 search engine

SEMKNOX GmbH

SiteSearch360
allow
SiteUpTimeBot monitoring

SiteUpTime

SiteUptime
allow
SkroutzBot social media

Skroutz S.A.

SkroutzBot
allow
Skype social media

Skype

SkypeUriPreview
allow
Slack Image Proxy social media

Slack

Slack-ImgProxy
allow
Slackbot other

Slack

Slackbot
allow
slopsearch-macos-crawler search engine
slopsearch-macos-crawler
allow
SmarshBot security

Smarsh

SmarshBot
allow
SMTnet PM Bot search engine

SMTnet

SMTnetPMBot
allow
Snipcart other

Snipcart

allow
Software Security Research security

Ruhr University Bochum

Software
allow
Sogou Push Spider search engine

Sogou

Sogou Push Spider
allow
Sogou web spider search engine

Sogou

allow
SokoCrawler search engine

eXperience Tech

SokoCrawler
allow
SolarWinds Observability monitoring

SolarWinds

SolarWinds Observability
allow
Spark Shipping other

Spark Shipping

SparkShipping
allow
SparkpostBot other

Bird

SparkPost
allow
Spawning AI ai crawler

Spawning

Spawning-AI
allow
special_archiver archiver

Internet Archive

special_archiver
allow
Spectate monitoring

Spectate

Spectate
allow
Splunk monitoring

Splunk

Rigor
allow
Splunk Attack Analyzer security

Splunk

TwinWaveScanner
allow
Splunk Synthetics monitoring

Splunk

Splunk Synthetics
allow
SSL Labs security

Qualys

allow
Stape Scanner monitoring

Stape Inc

Stape
allow
StatusCake monitoring

StatusCake

statuscake
allow
Steam Chat social media

Valve Software

SteamChat
allow
stepstoneCrawlBot search engine

StepStone

stepstoneCrawlBot
allow
StractBot search engine

Stract

StractBot
allow
Stripe other

Stripe

allow
stripe_apple_pay_verify_ownership other

Stripe

allow
Stripebot other

Stripe

Stripebot
allow
Sucuri security

Sucuri

Sucuri
allow
← Prev 7 / 9 Next →