Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
471 results
SiteLock
SiteLock (AccMan Connector)
SiteLock
SiteLockSpider
SEMKNOX GmbH
SiteSearch360
SiteUpTime
SiteUptime
Skroutz S.A.
SkroutzBot
Skype
SkypeUriPreview
Slack
Slack-ImgProxy
Slack
Slackbot
Smarsh
SmarshBot
SMTnet
SMTnetPMBot
Snipcart
Ruhr University Bochum
Software
Sogou
Sogou Push Spider
Sogou
eXperience Tech
SokoCrawler
SolarWinds
SolarWinds Observability
Spark Shipping
SparkShipping
Bird
SparkPost
Spawning
Spawning-AI
Internet Archive
special_archiver
Spectate
Spectate
Splunk
Rigor
Splunk
TwinWaveScanner
Splunk
Splunk Synthetics
Qualys
Stape Inc
Stape
StatusCake
statuscake
Valve Software
SteamChat
StepStone
stepstoneCrawlBot
Stract
StractBot
Stripe
Stripe
Stripe
Stripebot
Sucuri
Sucuri
Svix Inc.
sender-9YMgn
Swfiteq Ltd
Swifteq
DataDog
Synthetic
Telegram
TelegramBot
Ask
Teoma
Termly
TermlyBot
Ceramic
TerraCotta
Thousand Eyes
Thousand
TinEye
TinEye
TinEye
TinEye-bot
TinEye
TinEye-bot-live
TinEye
TinEye-Web
Bytedance
zhanzhang.toutiao.com
Trendiction S.A.
trendictionbot
Trustly Group AB
Automattic
Tumblr
Turnitin
Turnitin (https:
Twilio, Inc.
Splunk
TwinWaveScanner
Twitterbot
Updown
UpDownBot
Uptime
Uptime
GoDaddy
UptimeMonitoring
UptimeBot
UptimeBot
Uptimerobot
UptimeRobot
Uptimia
Uptimia