Skip to main content
Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

471 results

UptrendsBot monitoring

Uptrends GmbH

uptrends
allow
Usercentricsbot security

Usercentrics

Usercentricsbot
allow
VaultPress other

Automattic

VaultPress
allow
Verispider security

Project Honey Pot

Verispider
allow
VirusTotalScan security

VirusTotal

VirusTotalScan
allow
visionheight.com/scan security
visionheight.com/scan
allow
Watchbot monitoring

WatchBot

Watchbot
allow
Watchful monitoring

Watchful LLC

Watchful
allow
webpagetest monitoring

WebPageTest

webpagetest
allow
WebsitePulseBot monitoring

WebsitePulse

WebsitePulseBot
allow
Webstatus247 monitoring

WebStatus247

Webstatus247
allow
WebTotemBot security

WebTotem

WTotem
allow
WepchSearchEngine search engine

Wepch

WepchSearchEngine
allow
WkndHealthCheckBot monitoring

Wunderkind

wknd-bot
allow
WordCountBot social media

Weglot

WordCountBot
allow
Wordfence Central security

Wordfence

allow
Worldline Bot other

Worldline

allow
WorldPay other

WorldPay

allow
WormlyBot monitoring

Wormly

WormlyBot
allow
WOVN Crawler other

Wovn Technologies, Inc.

WOVN
allow
WP Time Capsule other

WP Time Capsule

allow
WP Umbrella monitoring

WP Umbrella

WPUmbrella
allow
WPMU DEV Broken Link Checker monitoring

WPMUDEV

WPMU
allow
WPMUDEV Uptime Monitor 5.0 monitoring

WPMUDEV

WPMUDEV Uptime Monitor 5.0 (https:
allow
WTotem security

WebTotem

WTotem
allow
WTotemBot security

WebTotem

WTotemBot
allow
XY-Archive-Compliance archiver

XY Archive Compliance

XY-Archive-Compliance
allow
XY-Archive-Compliance-Archiver archiver

XY Planning Network

XY-Archive-Compliance-Archiver
allow
Y!J-BRZ/YATSHA crawler search engine

Yahoo

allow
yacybot search engine

YaCy

yacybot
allow
Yahoo Link Preview social media

Yahoo

Yahoo Link Preview
allow
Yahoo Mail social media

Yahoo

YahooMailProxy
allow
Yahoo Slurp search engine

Yahoo

allow
Yahoo! Slurp search engine

Yahoo

allow
YahooCacheSystem social media

Yahoo

YahooCacheSystem
allow
YahooMailProxy social media

Yahoo

YahooMailProxy
allow
YandexBot search engine

Yandex

http://yandex.com/bots
allow rDNS
YandexImages search engine

Yandex

YandexImages
allow rDNS
YandexMobileBot search engine

Yandex

YandexMobileBot
allow rDNS
YandexRenderResourcesBot search engine

Yandex

YandexRenderResourcesBot
allow rDNS
YandexVertis search engine

Yandex

YandexVertis
allow rDNS
Yeti search engine

Naver

Yeti
allow
Yeti by Naver search engine

Naver

allow
YisouSpider search engine

Yisou

YisouSpider
allow
Yokoy Group Webhooks other

Yokoy Group AG

allow
YouBot search engine

You.com

YouBot
allow
Zapier other

Zapier Inc.

Zapier
allow
Zendesk Webhook other

Zendesk

allow
ZoomInfo search engine

ZoomInfo

ZoominfoBot
allow
ZumBot search engine

Zum Internet Corp

ZumBot
allow
Zvelo monitoring

zvelo

zvelo-bot
allow
← Prev 8 / 8 Next →