Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Site24x7
monitoring
Site24x7
site24x7
allow
SiteAuditBot
seo crawler
Semrush
SiteAuditBot
challenge
SiteBot
other
SiteBot
observe
Sitebulb
seo crawler
Sitebulb
Sitebulb
challenge
sitecheck.internetseer.com
other
sitecheck.internetseer.com
observe
SiteCheckerBotCrawler
seo crawler
Sitechecker
SiteCheckerBotCrawler
challenge
siteexplorer.info
other
siteexplorer.info
observe
SiteGuruCrawler
seo crawler
SiteGuru
SiteGuruCrawler
challenge
siteimprove
other
siteimprove
observe
Siteimprove Crawl
seo crawler
Siteimprove
SiteCheck-sitecrawl
challenge
siteimprove-lookup
other
siteimprove-lookup
observe
Siteimprove.com
other
Siteimprove.com
observe
Siteliner
seo crawler
Siteliner
Siteliner
challenge
SiteLock
security
SiteLock
SiteLock (AccMan Connector)
allow
SiteLockSpider
security
SiteLock
SiteLockSpider
allow
SitemapBot
other
SitemapBot
observe
SitemapCrawler
other
SitemapCrawler
observe
SiteSearch360
search engine
SEMKNOX GmbH
SiteSearch360
allow
SiteSnagger
other
SiteSnagger
observe
SitesOverPagesBot
other
SitesOverPagesBot
observe
SiteSucker
other
SiteSucker
observe
SiteUptime.com
other
SiteUptime.com
observe
SiteUpTimeBot
monitoring
SiteUpTime
SiteUptime
allow
Skroutz ImageBot
other
Skroutz S.A.
Skroutz
observe
SkroutzBot
social media
Skroutz S.A.
SkroutzBot
allow
Skype
social media
Skype
SkypeUriPreview
allow
SkyWatch
other
SkyWatch
observe
SlaccaleBot
other
SlaccaleBot
observe
Slack Image Proxy
social media
Slack
Slack-ImgProxy
allow
Slack-ImgProxy
ai assistant
Slack
Slack-ImgProxy
observe
Slackbot
other
Slack
Slackbot
allow
Slackbot-LinkExpanding
ai assistant
Slack
Slackbot-LinkExpanding
observe
SlackbotLinkExpanding
ai assistant
Slack
SlackbotLinkExpanding
observe
SleepBot
other
SleepBot
observe
Sleuth
other
Sleuth
observe
SlickBot
ai assistant
Slickstream
SlickBot
observe
Slickstream
seo crawler
Slickstream
SlickBot
challenge
slopsearch
ai assistant
Christopher Albert
slopsearch
observe
signed
slurp
other
slurp
observe
Smallwebindexbot
other
Smallwebindexbot
observe
SmarshBot
security
Smarsh
SmarshBot
allow
SmartBotCrawler
other
SmartBotCrawler
observe
SmartologyBot
seo crawler
Smartology
SmartologyBot
challenge
SMTBot
ai assistant
Similarweb
SMTBot
observe
SMTnet PM Bot
search engine
SMTnet
SMTnetPMBot
allow
Snap URL Preview Service
ai assistant
Snap
Snap
observe
Snapchat
ai assistant
Snapchat
Snapchat
observe
Snipcart
other
Snipcart
allow
SocialBeeAgent
other
SocialBeeAgent
observe
SocialRankIOBot
other
SocialRankIOBot
observe
Software Security Research
security
Ruhr University Bochum
Software
allow
Sogou
other
Sogou
observe
Sogou inst spider
other
observe
Sogou Push Spider
search engine
Sogou
Sogou Push Spider
allow
Sogou spider2
other
observe
Sogou web spider
search engine
Sogou
allow
SokoCrawler
search engine
eXperience Tech
SokoCrawler
allow
SolarWinds Observability
monitoring
SolarWinds
SolarWinds Observability
allow
Sonic
other
Sonic
observe
Sora Caisse POS
other
Sora Caisse POS
observe