Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

73 results · Social Media

Shortwave Image Fetcher social media

Shortwave Communications Inc.

Shortwave

allow

SkroutzBot social media

Skroutz S.A.

Skype

Slack Image Proxy social media

Slack

allow

Steam Chat social media

Valve Software

SteamChat

allow

Telegram Bot social media

Automattic

twitterbot social media

Twitter

Twitterbot

allow

WordCountBot social media

Weglot

WordCountBot

allow

Yahoo Link Preview social media

Yahoo

allow

Yahoo Mail social media

Yahoo

YahooMailProxy

allow

YahooCacheSystem social media

Yahoo

YahooCacheSystem

allow

YahooMailProxy social media

Yahoo

allow

← Prev 2 / 2 Next →