Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

75 results · AI Coding Agent

Trae ai coding agent

ByteDance

Trae

observe

updown.io ai coding agent

Updown

updown.io

observe

Uptime ai coding agent

Uptime

observe

uptimedoctor ai coding agent

uptimedoctor

uptimedoctor

observe

UptimeStatistics ai coding agent

UptimeStatistics

UptimeStatistics

observe

Validator.nu ai coding agent

W3C

Validator.nu

observe

W3C_CSS_Validator ai coding agent

W3C

W3C_CSS_Validator

observe

W3C_Validator ai coding agent

W3C

W3C_Validator

observe

W3C-checklink ai coding agent

W3C

W3C-checklink

observe

WebsiteOps ai coding agent

Watchful

WebsiteOps

observe

websitepulse ai coding agent

WebSitePulse

websitepulse

observe

WellKnownBot ai coding agent

Well-Known

WellKnownBot

observe

WP-Rocket-SaaS ai coding agent

WP Rocket

WPMU DEV

wpmudev ai coding agent

WPMUDEV

wpmudev

observe

← Prev 2 / 2 Next →