Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

73 results · Social Media

Shortwave Image Fetcher social media

Shortwave Communications Inc.

Shortwave

allow

SkroutzBot social media

Skroutz S.A.

Skype

Slack Image Proxy social media

Slack

Slack-ImgProxy

allow

Steam Chat social media

Valve Software

SteamChat

allow

Telegram Bot social media

Automattic

twitterbot social media

Twitter

Twitterbot

allow

WordCountBot social media

Weglot

WordCountBot

allow

Yahoo Link Preview social media

Yahoo

Yahoo Link Preview

allow

Yahoo Mail social media

Yahoo

YahooMailProxy

allow

YahooCacheSystem social media

Yahoo

YahooCacheSystem

allow

YahooMailProxy social media

Yahoo

YahooMailProxy

allow

← Prev 2 / 2 Next →