Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

812 results · Other

TaskMemoryBot other

TaskMemoryBot

TeleportPro other

Termly

theoldreader.com other

theoldreader.com

thinkers-bot other

TombaPublicWebCrawler other

TombaPublicWebCrawler

TopifyBot other

Toplistbot other

TopStoriesBot other

TopStoriesBot

ToutiaoSpider other

ToutiaoSpider

Traackr.com other

tracemyfile other

trafilatura other

Trapezohedron other

Trapezohedron

trendkite-akashic-crawler other

trendkite-akashic-crawler

trendybuzz other

trovitBot other

True_Robot other

TruliaBot other

Trustly Group AB

TweetedTimes other

twengabot other

Twilio Proxy other

Twilio, Inc.

UbiCrawler other

UGAResearchAgent other

UGAResearchAgent

UMichSearchEngineCrawler other

UMichSearchEngineCrawler

Updownerbot other

Uptime-Kuma other

Uptimebot.org other

Uptimebot.org

URL_Spider_Pro other

URL_Spider_Pro

urlappendbot other

UrlBeeBot other

urlsuma.de other

usasearch other

UsineNouvelleCrawler other

UsineNouvelleCrawler

UT-Dorkbot other

VaultPress other

Automattic

vebidoobot other

vecteurplus other

verticalsearch other

verticalsearch

← Prev 12 / 14 Next →