Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

812 results · Other

2checkout other

Verifone

360Spider-Image other

360Spider-Image

360Spider-Video other

360Spider-Video

A6-Indexer other

ABEvalBot other

absearch-crawler other

absearch-crawler

AcademicBotRTU other

AcademicBotRTU

Accessible Web Bot other

Accessible Web

adbeat-check-access-script other

adbeat-check-access-script

adequat-systems other

adequat-systems

ads-well-known-bot other

ads-well-known-bot

adscanner other

AdsTxtCrawler-CyberAgent other

AdsTxtCrawler-CyberAgent

Adyen

AfterShip Magento 2 Connector other

AfterShip

air.ai/scanning other

air.ai/scanning

AISearchBot other

Akismet

Amazon

AmiSoftware other

AntiSpam-Agent other

AntiSpam-Agent

AppInsights other

Apple App Site Association other

Apple

AppleNewsBot other

Aqua_Products other

Aqua_Products

asknread.com other

AspiegelBot other

ATSScraper-discovery other

ATSScraper-discovery

auramundi other

Authory

AwarioRssBot other

awesomecrawler other

awesomecrawler

BackDoorBot other

BacklinkBot other

← Prev 1 / 14 Next →