Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

497 results

KakaoTalk Scrap social media

Kakao

kakaotalk-scrap

allow

KargoBot-Artemis monitoring

Kargo

Kinsta

LegalMonster security

Legal Monster ApS

LegalMonster

allow

Let's Encrypt security

Let's Encrypt

allow

Lets Encrypt Validation Server security

Let's Encrypt

Let&

allow

Level9SearchBot search engine

Level 9

Level9SearchBot

allow

Linespider search engine

LINE Corporation

Linespider

allow

Link Tiger monitoring

LinkTiger, Inc.

LinkTiger

allow

LinkedInBot social media

LinkedInBot

allow

LinksIndexerBot social media

Links Indexer

LinksIndexerBot

allow

logicmonitor monitoring

LogicMonitor

LogicMonitor InternalService

allow

LohiSoftBot search engine

LohiSoft

LohiSoftBot

allow

Loomly Bot social media

Loomly

LoomlyBot

allow

Luxxlebot search engine

Luxxle

Lyonl

lyonl-crawler search engine

Lyonl

lyonl-crawler

allow

Macrobondbot monitoring

Macrobondbot

Macrobondbot

allow

MagiBot search engine

Peak Labs

MagiBot

allow

MailRUBot social media

Mail Russia

Mail.RU_Bot

allow

MainWP other

Direct Support / Website Managed

Make

ManageWP

Marginalia Search search engine

Marginalia

search.marginalia.nu

allow

Mars Finder search engine

Mars Flag

Mars

allow

Mavifinds Bot monitoring

Mavifinds

Mavifinds

allow

mediaboardbot monitoring

Mediaboard

mediaboardbot

allow

Medialogia Bot monitoring

Medialogia

MedialogiaBot

allow

MerchantSecurityScanner security

Stripe

MerchantSecurityScanner

allow

Meta-WebIndexer social media