Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

2,114 results

AgentRadar-Research other

Boston Consulting Group

AgentRadar-Research

observe

AgentReadinessScanner ai coding agent

Cloudflare

AgentReadinessScanner

observe

AGI Agent ai assistant

AGI, Inc

agi-agent-2

observe signed

Ahrefs Site Audit seo crawler

Ahrefs

AhrefsSiteAudit

challenge

AhrefsBot seo crawler

Ahrefs

AhrefsBot

challenge

AI Search ai crawler

Cloudflare

Cloudflare-AI-Search

Ai2

AI2Bot-DeepResearchEval ai assistant

Ai2

AI2Bot-DeepResearchEval

observe

Ai2Bot-Dolma ai scraper

Ai2

Ai2Bot-Dolma

observe

aiHitBot ai assistant

aiHit

air.ai/scanning other

AITrustRadar security

AI Trust Radar

AITrustRadar

allow

AIWebIndex ai data provider

Lyrenth

AIWebIndex

observe

AIWebIndex-Agent ai crawler

Lyrenth

Akismet

Alertsite by Smartbear monitoring

Smartbear

AlertSite

allow

Alexa Archive search engine

Alexandria.org

Alexa

allow

alexa site audit search engine

Amazon

AlexandriaOrgBot search engine

Alexandria.org

Algolia search engine

Algolia

Algolia

allow

AliyunSecBot security

Alibaba

AliyunSecBot

allow

All Africa Crawler search engine

AllAfrica Global Media

AllAfrica

allow

AllAfrica NewsBot ai assistant

AllAfrica

observe

Alli AI Bot seo crawler

Alli AI

challenge

allOrigins ai assistant

Amazing-SearchBot ai assistant

Amazing

Amazing-SearchBot

observe

AmazingTest-SearchBot ai assistant

Amazing

AmazingTest-SearchBot

observe

Amazon AdBot seo crawler

Amazon

AmazonAdBot

allow rDNS

Amazon Bedrock AgentCore Browser other

Amazon

Amazon Bedrock AgentCore Browser

observe

Amazon Bedrock AgentCore Browser (AP Northeast) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-northeast-1

observe signed

Amazon Bedrock AgentCore Browser (AP South) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-south-1

observe signed

Amazon Bedrock AgentCore Browser (AP Southeast 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-southeast-1

observe signed

Amazon Bedrock AgentCore Browser (AP Southeast) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-southeast-2

observe signed

Amazon Bedrock AgentCore Browser (EU Central 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-eu-central-1

observe signed

Amazon Bedrock AgentCore Browser (EU West 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-eu-west-1

observe signed

Amazon Bedrock AgentCore Browser (US East 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-east

observe signed

Amazon Bedrock AgentCore Browser (US East 2) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-east-2

observe signed

Amazon Bedrock AgentCore Browser (US West 2) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-west

observe signed

Amazon Buy For Me ai assistant

Amazon

AmazonBuyForMe

observe

Amazon Contxbot seo crawler

Amazon

contxbot

challenge

Amazon Kendra ai crawler

Amazon

amazon-kendra

observe

Amazon-Advertising-ad-standards-bot ai assistant

Amazon

Amazon-Advertising-ad-standards-bot

observe

amazon-QBusiness ai assistant

Amazon

Amazon

AmazonProductDiscovery ai assistant

Amazon

AmazonProductDiscovery

Amzn-SearchBot ai crawler

Amazon

Amzn-SearchBot

observe

Amzn-User ai assistant

Amazon

Amzn-User

observe

Anchor Browser ai crawler

Anchor

Anchor

observe signed

AndersPinkBot ai assistant

Anders Pink

Andi

Direqt

ant-html ai assistant

Kuaishou

ant-html

observe

anthropic-ai ai agent

Anthropic

anthropic-ai

observe

← Prev 2 / 36 Next →