Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

007ac9 Crawler other

2ip bot ai coding agent

2IP

360Spider-Image other

360Spider-Image

360Spider-Video other

360Spider-Video

A Patent Crawler other

A360-Search ai assistant

Area360

A6-Indexer other

AASA-Bot ai assistant

Apple

ABEvalBot other

absearch-crawler other

absearch-crawler

AcademicBotRTU other

AcademicBotRTU

Acunetix Security Scanner other

Acunetix Web Vulnerability Scanner other

adbeat_bot ai assistant

Adbeat

adbeat-check-access-script other

adbeat-check-access-script

AddThis.com ai assistant

Oracle

adequat-systems other

adequat-systems

AdkernelTopicCrawler ai assistant

AdKernel

AdkernelTopicCrawler

admantx ai assistant

Integral Ad Science

AdmixerBot ai assistant

Admixer

AdpResearchBot ai assistant

ADP

AdpResearchBot

ads-well-known-bot other

ads-well-known-bot

adscanner other

AdsNexusBot other

adstxt-research other

adstxt-research

AdsTxtCrawler-CyberAgent other

AdsTxtCrawler-CyberAgent

AffsignalCrawler ai assistant

Affsignal

AffsignalCrawler

AgencyDiscoveryCrawler other

AgencyDiscoveryCrawler

AgentDataBot ai data provider

AgentData

AgentRadar-Research other

Boston Consulting Group

AgentRadar-Research

AgentReadinessScanner ai coding agent

Cloudflare

AgentReadinessScanner

AGI Agent ai assistant

AGI, Inc

AI Search ai crawler

Cloudflare

Cloudflare-AI-Search

AI2Bot ai scraper

Ai2

AI2Bot-DeepResearchEval ai assistant

Ai2

AI2Bot-DeepResearchEval

Ai2Bot-Dolma ai scraper

Ai2

aiHitBot ai assistant

aiHit

air.ai/scanning other

air.ai/scanning

AISearchBot other

AIWebIndex ai data provider

Lyrenth

AIWebIndex-Agent ai crawler

Lyrenth

AIWebIndex-Agent

AllAfrica NewsBot ai assistant

AllAfrica

allOrigins ai assistant

Amazing-SearchBot ai assistant

Amazing

Amazing-SearchBot

AmazingTest-SearchBot ai assistant

Amazing

AmazingTest-SearchBot

← Prev 1 / 24 Next →