Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
SimpleScraper
other
SimpleScraper
observe
Sindup
other
Sindup
observe
SirdataBot
other
Sirdata
SirdataBot
observe
sistrix crawler
other
sistrix
observe
site-scraper-service
other
site-scraper-service
observe
SiteBot
other
SiteBot
observe
sitecheck.internetseer.com
other
sitecheck.internetseer.com
observe
siteexplorer.info
other
siteexplorer.info
observe
siteimprove
other
siteimprove
observe
siteimprove-lookup
other
siteimprove-lookup
observe
Siteimprove.com
other
Siteimprove.com
observe
SitemapBot
other
SitemapBot
observe
SitemapCrawler
other
SitemapCrawler
observe
SiteSnagger
other
SiteSnagger
observe
SitesOverPagesBot
other
SitesOverPagesBot
observe
SiteStatsBot
ai assistant
Site Stats Database
SiteStatsBot
observe
SiteSucker
other
SiteSucker
observe
SiteUptime.com
other
SiteUptime.com
observe
SkillGymCorpusBot
other
SkillGymCorpusBot
observe
Skroutz ImageBot
other
Skroutz S.A.
Skroutz
observe
SkynetBot
other
SkynetBot
observe
SkytabBot
other
SkytabBot
observe
SkyWatch
other
SkyWatch
observe
SlaccaleBot
other
SlaccaleBot
observe
Slack-ImgProxy
ai assistant
Slack
Slack-ImgProxy
observe
Slackbot-LinkExpanding
ai assistant
Slack
Slackbot-LinkExpanding
observe
SleepBot
other
SleepBot
observe
Sleuth
other
Sleuth
observe
SlickBot
ai assistant
Slickstream
SlickBot
observe
slopsearch
ai assistant
Christopher Albert
slopsearch
observe
signed
slurp
other
slurp
observe
Smallwebindexbot
other
Smallwebindexbot
observe
SmartBotCrawler
other
SmartBotCrawler
observe
SMBsBot
ai assistant
SMBsBot
observe
SMTBot
ai assistant
Similarweb
SMTBot
observe
Snap URL Preview Service
ai assistant
Snap
Snap
observe
Snapchat
ai assistant
Snapchat
Snapchat
observe
SocialBeeAgent
other
SocialBeeAgent
observe
SocialRankIOBot
other
SocialRankIOBot
observe
SofyaBot
ai data provider
Sofya
SofyaBot
observe
Sogou
other
Sogou
observe
Sogou inst spider
other
observe
Sogou spider2
other
observe
SomethinggBot
ai agent
somethingg
SomethinggBot
observe
Sonic
other
Sonic
observe
Sora Caisse POS
other
Sora Caisse POS
observe
Sosospider
other
Sosospider
observe
Sottopop
ai assistant
UpContent
Sottopop
observe
Sottopop-user-rss
ai assistant
UpContent
Sottopop-user-rss
observe
SottopopNone
ai assistant
UpContent
SottopopNone
observe
SpamCipherBot
ai assistant
SpamCipher
SpamCipherBot
observe
SpankBot
other
SpankBot
observe
spanner
other
spanner
observe
spbot
other
spbot
observe
SpeedyIndexBot
other
SpeedyIndexBot
observe
Speroll-AdsTxt-Crawler
ai assistant
Speroll Media
Speroll-AdsTxt-Crawler
observe
Spider
ai scraper
Spider
Spider
observe
SpiderLing
other
SpiderLing
observe
Spinn3r
other
Spinn3r
observe
spotter
other
spotter
observe