Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

1,284 results

Tiny ai assistant

Tiny Tiny RSS

TLS tester other

TombaPublicWebCrawler other

TombaPublicWebCrawler

TongyiBot ai assistant

Alibaba

TopifyBot other

Toplistbot other

TopStoriesBot other

TopStoriesBot

ToutiaoSpider other

ToutiaoSpider

Traackr.com other

tracemyfile other

Trae ai coding agent

ByteDance

trafilatura other

Trapezohedron other

Trapezohedron

trendkite-akashic-crawler other

trendkite-akashic-crawler

TrendsmapResolver ai assistant

Trendsmap

TrendsmapResolver

trendybuzz other

trovitBot other

True_Robot other

TruliaBot other

TurnitinBot ai scraper

Turnitin

TweetedTimes other

TweetmemeBot ai assistant

Meltwater

twengabot other

TwinAgent ai assistant

Twin

Twingly ai assistant

Twingly

Twingly Recon-Klondike

Twingly Recon-Sjostrom ai assistant

Twingly

UbiCrawler other

UGAResearchAgent other

UGAResearchAgent

uipbot ai assistant

Semasio

um-FC ai assistant

Ubermetrics

um-IC ai assistant

Unicepta

um-LN ai assistant

Unicepta

UMichSearchEngineCrawler other

UMichSearchEngineCrawler

UpContent ai assistant

UpContent

UpContent-QA ai assistant

UpContent

UpContent-Test ai assistant

UpContent

UpContent-Test

updown.io ai coding agent

Updown

updown.io daemon ai coding agent

updown.io

Updownerbot other

Upstash QStash ai assistant

Upstash

Upstash-QStash

Uptime-Kuma other

Uptimebot.org other

Uptimebot.org

uptimedoctor ai coding agent

uptimedoctor

UptimeStatistics ai coding agent

UptimeStatistics

UptimeStatistics

URL Control other

URL_Spider_Pro other

URL_Spider_Pro

urlappendbot other

UrlBeeBot other

urlsuma.de crawler other

URLy Warning other

usasearch other

UseAI ai assistant

Use AI

← Prev 19 / 22 Next →