Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
theoldreader.com
other
theoldreader.com
observe
TheSkinnyAI
ai assistant
TheSkinnyCrawler
observe
signed
TheWebReport
ai assistant
TheWebReport
observe
Thinkbot
other
Thinkbot
observe
thinkers-bot
other
thinkers-bot
observe
Thinklab
other
Thinklab
observe
Thousand Eyes Cloud Agent
monitoring
Thousand Eyes
Thousand
allow
TigerBot
other
TigerBot
observe
TikTokSpider
ai assistant
ByteDance
TikTokSpider
observe
Timpibot
ai scraper
Timpi
Timpibot
observe
TinEye
search engine
TinEye
TinEye
allow
TinEye-bot
search engine
TinEye
TinEye-bot
allow
TinEye-bot-live
search engine
TinEye
TinEye-bot-live
allow
TinEye-Web
search engine
TinEye
TinEye-Web
allow
Tiny
ai assistant
Tiny Tiny RSS
Tiny
observe
Titan
other
Titan
observe
TitanBot
ai assistant
Titan Materials Corp
TitanBot
observe
TLDWPBot
other
TLDWPBot
observe
TLS tester
other
observe
toCrawl
other
toCrawl
observe
TombaPublicWebCrawler
other
TombaPublicWebCrawler
observe
TongyiBot
ai assistant
Alibaba
TongyiBot
observe
topicsearch-crawler
ai assistant
FatGrid
topicsearch-crawler
observe
TopifyBot
other
TopifyBot
observe
Toplistbot
other
Toplistbot
observe
TopStoriesBot
other
TopStoriesBot
observe
Toutiao
search engine
Bytedance
zhanzhang.toutiao.com
allow
ToutiaoSpider
other
ToutiaoSpider
observe
tphotobot
ai assistant
EstiDraft
tphotobot
observe
Traackr.com
other
Traackr.com
observe
tracemyfile
other
tracemyfile
observe
Trae
ai coding agent
ByteDance
Trae
observe
trafilatura
other
trafilatura
observe
Trapezohedron
other
Trapezohedron
observe
Trellis-Services
seo crawler
Mediavine
Trellis-Services
challenge
trendeo
other
trendeo
observe
Trendiction Bot
monitoring
Trendiction S.A.
trendictionbot
allow
trendkite-akashic-crawler
other
trendkite-akashic-crawler
observe
TrendsmapResolver
ai assistant
Trendsmap
TrendsmapResolver
observe
trendybuzz
other
trendybuzz
observe
trovitBot
other
trovitBot
observe
True_Robot
other
True_Robot
observe
TruliaBot
other
TruliaBot
observe
Trustly
other
Trustly Group AB
allow
TSMbot
other
TSMbot
observe
TTD Content
seo crawler
The Trade Desk
TTD-Content
challenge
Tumblr
social media
Automattic
Tumblr
allow
turingos
other
turingos
observe
Turnitin
archiver
Turnitin
Turnitin (https:
allow
TurnitinBot
ai scraper
Turnitin
TurnitinBot
observe
TweetedTimes
other
TweetedTimes
observe
TweetmemeBot
ai assistant
Meltwater
TweetmemeBot
observe
twengabot
other
twengabot
observe
Twilio Proxy
other
Twilio, Inc.
allow
TwinAgent
ai assistant
Twin
TwinAgent
observe
Twingly
ai assistant
Twingly
Twingly Recon-Klondike
observe
Twingly Recon-Sjostrom
ai assistant
Twingly
observe
TwinWaveScanner
security
Splunk
TwinWaveScanner
allow
twitterbot
social media
Twitterbot
allow
Twurly
other
Twurly
observe