Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
tracemyfile
other
tracemyfile
observe
Trae
ai coding agent
ByteDance
Trae
observe
trafilatura
other
trafilatura
observe
Trapezohedron
other
Trapezohedron
observe
Trellis-Services
seo crawler
Mediavine
Trellis-Services
challenge
trendeo
other
trendeo
observe
Trendiction Bot
monitoring
Trendiction S.A.
trendictionbot
allow
trendkite-akashic-crawler
other
trendkite-akashic-crawler
observe
TrendsmapResolver
ai assistant
Trendsmap
TrendsmapResolver
observe
trendybuzz
other
trendybuzz
observe
trovitBot
other
trovitBot
observe
True_Robot
other
True_Robot
observe
TruliaBot
other
TruliaBot
observe
Trustly
other
Trustly Group AB
allow
TSMbot
other
TSMbot
observe
TTD Content
seo crawler
The Trade Desk
TTD-Content
challenge
Tumblr
social media
Automattic
Tumblr
allow
turingos
other
turingos
observe
Turnitin
archiver
Turnitin
Turnitin
allow
TurnitinBot
ai scraper
Turnitin
TurnitinBot
observe
TweetedTimes
other
TweetedTimes
observe
TweetmemeBot
ai assistant
Meltwater
TweetmemeBot
observe
twengabot
other
twengabot
observe
Twilio Proxy
other
Twilio, Inc.
allow
TwinAgent
ai assistant
Twin
TwinAgent
observe
Twingly
ai assistant
Twingly
Twingly
observe
TwinWaveScanner
security
Splunk
TwinWaveScanner
allow
twitterbot
social media
Twitterbot
allow
Twurly
other
Twurly
observe
UbiCrawler
other
UbiCrawler
observe
UGAResearchAgent
other
UGAResearchAgent
observe
uipbot
ai assistant
Semasio
uipbot
observe
um-FC
ai assistant
Ubermetrics
um-FC
observe
um-IC
ai assistant
Unicepta
um-IC
observe
um-LN
ai assistant
Unicepta
um-LN
observe
UMichSearchEngineCrawler
other
UMichSearchEngineCrawler
observe
UpContent
ai assistant
UpContent
UpContent
observe
UpContent-QA
ai assistant
UpContent
UpContent-QA
observe
UpContent-Test
ai assistant
UpContent
UpContent-Test
observe
upday
seo crawler
upday GmbH & Co. KG
upday
challenge
updown.io
ai coding agent
Updown
updown.io
observe
UpDownBot
monitoring
Updown
UpDownBot
allow
Updownerbot
other
Updownerbot
observe
Upflow
other
Upflow
observe
Upstash QStash
ai assistant
Upstash
observe
signed
Uptime
ai coding agent
Uptime
observe
Uptime LLC
monitoring
Uptime
Uptime
allow
Uptime Monitoring
monitoring
GoDaddy
allow
Uptime-Kuma
other
Uptime-Kuma
observe
UptimeBot
monitoring
UptimeBot
UptimeBot
allow
Uptimebot.org
other
Uptimebot.org
observe
uptimedoctor
ai coding agent
uptimedoctor
uptimedoctor
observe
UptimeRobot
monitoring
Uptimerobot
UptimeRobot
allow
UptimeStatistics
ai coding agent
UptimeStatistics
UptimeStatistics
observe
Uptimia
monitoring
Uptimia
Uptimia
allow
UptrendsBot
monitoring
Uptrends GmbH
uptrends
allow
URL
other
URL
observe
URL_Spider_Pro
other
URL_Spider_Pro
observe
urlappendbot
other
urlappendbot
observe
UrlBeeBot
other
UrlBeeBot
observe