Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

IncumbINTBot ai assistant

Reticle

IndeedBot other

INETDEX-Bot other

InfoNaviRobot other

InfoNaviRobot

Inoreader ai assistant

Inoreader

inoreader.com ai assistant

Innologica

inoreader.com

Instapaper other

Instant Paper

Instaparser ai assistant

Instapaper

Instinct ai agent

Instinct

integromedb other

intelium_bot other

IntentBeamBot ai assistant

IntentBeamBot

InteractiveAdvertisingBureau ai assistant

OneTag

InteractiveAdvertisingBureau

InterfaxScanBot other

InterfaxScanBot

InterviewOS ai assistant

Subspace

IntricleBot ai assistant

IonCrawl ai assistant

IONOS

ip-web-crawler.com other

ip-web-crawler.com

ips-agent other

IronFountain-Leads other

Iron Fountain

IronFountain-Leads

IsonSearchBot ai crawler

IsonAI

IsonSearchBot

istellabot other

it2media-domain-crawler other

it2media-domain-crawler

Jamie's Spider other

JenkersBot other

jetmon ai coding agent

Automattic

JikeSpider other

JobboerseBot other

JobHuntBot other

jobs-crawler other

jpg-newsbot other

Jugendschutzprogramm-Crawler ai assistant

JusProg

Jugendschutzprogramm-Crawler

kagi-fetcher ai assistant

Kagi

kaikki.org-digital-archive other

kaikki.org-digital-archive

KamenX ai coding agent

Kangaroo Bot ai scraper

Kangaroo LLM

KargoBot ai assistant

Kargo

KargoBot-Artemis-Mobile ai assistant

Kargo

KargoBot-Artemis-Mobile

kb.dk_bot ai scraper

Netarkivet

Keenable-User ai data provider

Keenable

Keenable-User

KeenableBot ai data provider

Keenable

Kenjin Spider other

Kernel ai assistant

Kernel

Keydrop.io other

← Prev 11 / 24 Next →