Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

290 results · AI Assistant

A360-Search ai assistant

Area360

A360-Search

observe

AASA-Bot ai assistant

Apple

AASA-Bot

observe

ActiveComply ai assistant

ActiveComply

ActiveComply

observe

adbeat_bot ai assistant

Adbeat

adbeat_bot

observe

AddThis.com ai assistant

Oracle

AddThis.com

observe

AdkernelTopicCrawler ai assistant

AdKernel

AdkernelTopicCrawler

Integral Ad Science

AdmixerBot ai assistant

Admixer

AdmixerBot

observe

AdsBot-Google ai assistant

Google

AdsBot-Google

observe

AdsBot-Google-Mobile ai assistant

Google

AdsBot-Google-Mobile

observe rDNS

AffsignalCrawler ai assistant

Affsignal

AffsignalCrawler

observe

AGI Agent ai assistant

AGI, Inc

agi-agent-2

observe signed

AI2Bot-DeepResearchEval ai assistant

Ai2

AI2Bot-DeepResearchEval

observe

aiHitBot ai assistant

aiHit

aiHitBot

observe

AllAfrica ai assistant

AllAfrica Global Media

AllAfrica

observe

AllAfrica NewsBot ai assistant

AllAfrica

observe

Amazing-SearchBot ai assistant

Amazing

Amazing-SearchBot

observe

AmazingTest-SearchBot ai assistant

Amazing

AmazingTest-SearchBot

observe

Amazon Bedrock AgentCore Browser (AP Northeast) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-northeast-1

observe signed

Amazon Bedrock AgentCore Browser (AP South) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-south-1

observe signed

Amazon Bedrock AgentCore Browser (AP Southeast 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-southeast-1

observe signed

Amazon Bedrock AgentCore Browser (AP Southeast) ai assistant

Amazon

amazon-bedrock-agentcore-browser-ap-southeast-2

observe signed

Amazon Bedrock AgentCore Browser (EU Central 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-eu-central-1

observe signed

Amazon Bedrock AgentCore Browser (EU West 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-eu-west-1

observe signed

Amazon Bedrock AgentCore Browser (US East 1) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-east

observe signed

Amazon Bedrock AgentCore Browser (US East 2) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-east-2

observe signed

Amazon Bedrock AgentCore Browser (US West 2) ai assistant

Amazon

amazon-bedrock-agentcore-browser-us-west

observe signed

Amazon Buy For Me ai assistant

Amazon

AmazonBuyForMe

observe

Amazon-Advertising-ad-standards-bot ai assistant

Amazon

Amazon-Advertising-ad-standards-bot

observe

amazon-QBusiness ai assistant

Amazon

amazon-QBusiness

observe

AmazonProductDiscovery ai assistant

Amazon

AmazonProductDiscovery

observe

Amzn-User ai assistant

Amazon

Amzn-User

observe

AndersPinkBot ai assistant

Anders Pink

AndersPinkBot

observe

Apify Website Content Crawler ai assistant

Apify

ApifyWebsiteContentCrawler

observe signed

ArchiveBot ai assistant

Wikimedia

ArchiveBot

observe

ArenaUnfurlBot ai assistant

Arena

ArenaUnfurlBot

observe

Assaybot ai assistant

Index Exchange

Assaybot

observe

AudigentAdBot ai assistant

Audigent

AudigentAdBot

observe

Automattic Analytics Crawler ai assistant

WordPress

Automattic

observe

AwarioBot ai assistant

Awario

AwarioBot

observe

BestChange ai assistant

BestChange

BestChange

observe

bigsur.ai ai assistant

Big Sur AI

bigsur.ai

observe

bitlybot ai assistant

Bitly

bitlybot

observe

BitSightBot ai assistant

Bitsight

BitSightBot

observe

Blackboard ai assistant

Anthology

Blackboard

observe

BlingERP ai assistant

Bling

BlingERP

observe

Bloglines ai assistant

Ask

Bluesky

BomboraBot ai assistant

Bombora

BomboraBot

observe

BrandVerity ai assistant

BrandVerity

BrandVerity

observe

Browserbase ai assistant

Browserbase

Browserbase

observe signed

BubblesBot ai assistant

Bubbles

BubblesBot

observe

BublupBot ai assistant

Bublup

Hypefactors

Buttondown ai assistant

Buttondown

Buttondown

observe

CartAI Agentic Commerce as a Service ai assistant

CartAI

cartai-agentic-commerce-as-a-service

observe signed

channable ai assistant

Channable

channable

observe

ChatGPT agent ai assistant

OpenAI

ChatGPT

observe signed

ChatGPT-User ai assistant

OpenAI

ChatGPT-User

observe

CheckMarkNetwork ai assistant

CheckMark Network

CheckMarkNetwork

observe

← Prev 1 / 5 Next →