Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
21 results · Archiver
Archive-It
archiver
Internet Archive
Archive-It
allow
Arquivo Web Crawler
archiver
Arquivo
Arquivo-web-crawler
allow
Arquivo-web-crawler
archiver
Arquivo
Arquivo-web-crawler
allow
bl.uk_lddc_bot
archiver
British Library
bl.uk_lddc_bot
allow
bne.es_bot
archiver
bne.es_bot
allow
bnf.fr_bot
archiver
Bibliothèque nationale de France
bnf.fr_bot
allow
CUCH.org Archive Bot
archiver
C U Cyber History
CUCH.org
allow
heritrix
archiver
Internet Archive
heritrix
allow
ia_archiver
archiver
Internet Archive
ia_archiver
allow
ia_archiver-web.archive.org
archiver
Internet Archive
ia_archiver-web.archive.org
allow
IABot
archiver
Wikimedia Foundation
IABot
allow
Internet Archive
archiver
Internet Archive
archive.org_bot
allow
Internet Archive - Archive-It
archiver
Archive-It
Archive-It
allow
mirrorweb
archiver
MirrorWeb
mirrorweb
allow
netarkivindsamling
archiver
Netarkivet
netarkivindsamling
allow
NewsBank
archiver
NewsBank
NewsBank
allow
Nicecrawler
archiver
NiceCrawler
Nicecrawler
allow
special_archiver
archiver
Internet Archive
special_archiver
allow
Turnitin
archiver
Turnitin
Turnitin (https:
allow
XY-Archive-Compliance
archiver
XY Archive Compliance
XY-Archive-Compliance
allow
XY-Archive-Compliance-Archiver
archiver
XY Planning Network
XY-Archive-Compliance-Archiver
allow