Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
Thinkbot
other
Thinkbot
observe
thinkers-bot
other
thinkers-bot
observe
Thinklab
other
Thinklab
observe
TigerBot
other
TigerBot
observe
TikTokSpider
ai assistant
ByteDance
TikTokSpider
observe
Timpibot
ai scraper
Timpi
Timpibot
observe
Tiny
ai assistant
Tiny Tiny RSS
Tiny
observe
Titan
other
Titan
observe
TitanBot
ai assistant
Titan Materials Corp
TitanBot
observe
TLDWPBot
other
TLDWPBot
observe
TLS tester
other
observe
toCrawl
other
toCrawl
observe
TombaPublicWebCrawler
other
TombaPublicWebCrawler
observe
TongyiBot
ai assistant
Alibaba
TongyiBot
observe
topicsearch-crawler
ai assistant
FatGrid
topicsearch-crawler
observe
TopifyBot
other
TopifyBot
observe
Toplistbot
other
Toplistbot
observe
TopStoriesBot
other
TopStoriesBot
observe
ToutiaoSpider
other
ToutiaoSpider
observe
tphotobot
ai assistant
EstiDraft
tphotobot
observe
Traackr.com
other
Traackr.com
observe
tracemyfile
other
tracemyfile
observe
Trae
ai coding agent
ByteDance
Trae
observe
trafilatura
other
trafilatura
observe
Trapezohedron
other
Trapezohedron
observe
trendeo
other
trendeo
observe
trendkite-akashic-crawler
other
trendkite-akashic-crawler
observe
TrendsmapResolver
ai assistant
Trendsmap
TrendsmapResolver
observe
trendybuzz
other
trendybuzz
observe
trovitBot
other
trovitBot
observe
True_Robot
other
True_Robot
observe
TruliaBot
other
TruliaBot
observe
TSMbot
other
TSMbot
observe
turingos
other
turingos
observe
TurnitinBot
ai scraper
Turnitin
TurnitinBot
observe
TweetedTimes
other
TweetedTimes
observe
TweetmemeBot
ai assistant
Meltwater
TweetmemeBot
observe
twengabot
other
twengabot
observe
TwinAgent
ai assistant
Twin
TwinAgent
observe
Twingly
ai assistant
Twingly
Twingly Recon-Klondike
observe
Twingly Recon-Sjostrom
ai assistant
Twingly
observe
Twurly
other
Twurly
observe
UbiCrawler
other
UbiCrawler
observe
UGAResearchAgent
other
UGAResearchAgent
observe
uipbot
ai assistant
Semasio
uipbot
observe
um-FC
ai assistant
Ubermetrics
um-FC
observe
um-IC
ai assistant
Unicepta
um-IC
observe
um-LN
ai assistant
Unicepta
um-LN
observe
UMichSearchEngineCrawler
other
UMichSearchEngineCrawler
observe
UpContent
ai assistant
UpContent
UpContent
observe
UpContent-QA
ai assistant
UpContent
UpContent-QA
observe
UpContent-Test
ai assistant
UpContent
UpContent-Test
observe
updown.io
ai coding agent
Updown
updown.io
observe
updown.io daemon
ai coding agent
updown.io
observe
Updownerbot
other
Updownerbot
observe
Upflow
other
Upflow
observe
Upstash QStash
ai assistant
Upstash
Upstash-QStash
observe
signed
Uptime-Kuma
other
Uptime-Kuma
observe
Uptimebot.org
other
Uptimebot.org
observe
uptimedoctor
ai coding agent
uptimedoctor
uptimedoctor
observe