Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
SMTnet
SMTnetPMBot
Snap
Snap
Snapchat
Snapchat
Snipcart
SocialBeeAgent
SocialRankIOBot
Ruhr University Bochum
Software
Sofya
SofyaBot
Sogou
Sogou
Sogou Push Spider
Sogou
eXperience Tech
SokoCrawler
SolarWinds
SolarWinds Observability
somethingg
SomethinggBot
Sonic
Sora Caisse POS
Sosospider
UpContent
Sottopop
UpContent
Sottopop-user-rss
UpContent
SottopopNone
SpamCipher
SpamCipherBot
SpankBot
spanner
Spark Shipping
SparkShipping
Bird
SparkPost
Spawning
Spawning-AI
spbot
Internet Archive
special_archiver
Spectate
Spectate
SpeedyIndexBot
Speroll Media
Speroll-AdsTxt-Crawler
Spider
Spider
SpiderLing
Spinn3r
Splunk
Rigor
Splunk
TwinWaveScanner
Splunk
Splunk Synthetics
spotter
SpringserveBot
SputnikBot
SpyglassesBot
Spyglasses
SpyglassesBot
Open-source
SQWatcher
Qualys
Stape Inc
Stape
star-finder.de
Start.me
StartmeBot
StataCorp LLC
Statabot
Statistik Bot
StatistikAustria
StatsDrone
StatsDroneBot
StatusCake
statuscake
StatusNest
StatusNestBacklinkSpider
Valve Software
SteamChat
StepStone
stepstoneCrawlBot
Bryan Norcross
StormIntelCrawler
StorygizeBot
Stract
StractBot