Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

xAI-Grok ai agent

xAI

xAI-Grok

observe

xAI-SearchBot ai crawler

xAI

Xenu Link Sleuth other

Xenu's Link Sleuth 1.1c other

XML Sitemaps Generator other

XY Archive Compliance Bot other

XY Archive Compliance

XY-Archive-Compliance

YaDirectFetcher ai assistant

Yandex

YaDirectFetcher

observe

Yahoo Pipes 1.0 other

Linkfluence

YandexAdditional ai crawler

Yandex

YandexAdditional

observe

YandexAdditionalBot ai crawler

Yandex

YandexAdditionalBot

Yellowbrandprotectionbot other

Yellowbrandprotectionbot

Yelp

Ahrefs

YGS Group Falconer Scraper ai crawler

01.AI

YiyanBot ai assistant

Baidu

ZanistaBot ai crawler

Zanista

Zenphony

Zeus Link Scout other

ZipchatBot ai assistant

Zipchat Inc

ZoteroTranslationServer ai assistant

Wikimedia

ZoteroTranslationServer

← Prev 24 / 24 Next →