Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

908 results · Other

Verifone

360Spider-Image other

360Spider-Image

observe

360Spider-Video other

A Patent Crawler other

absearch-crawler other

Accessible Web Bot other

Accessible Web

Acunetix Security Scanner other

observe

Acunetix Web Vulnerability Scanner other

observe

adbeat-check-access-script other

adbeat-check-access-script

adequat-systems other

adequat-systems

observe

ads-well-known-bot other

adstxt-research other

adstxt-research

observe

AdsTxtCrawler-CyberAgent other

AdsTxtCrawler-CyberAgent

Adyen

AfterShip Magento 2 Connector other

AfterShip

allow

AgencyDiscoveryCrawler other

AgencyDiscoveryCrawler

observe

AgentRadar-Research other

Boston Consulting Group

AgentRadar-Research

air.ai/scanning other

Akismet

Amazon Bedrock AgentCore Browser other

Amazon

Amazon Bedrock AgentCore Browser

Apple App Site Association other

Apple

ATSScraper-discovery other

ATSScraper-discovery

← Prev 1 / 16 Next →