Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
21 results · Archiver
Archive-It
archiver
Internet Archive
Archive-It
allow
Arquivo Web Crawler
archiver
Arquivo
allow
Arquivo-web-crawler
archiver
Arquivo
Arquivo-web-crawler
allow
bl.uk_lddc_bot
archiver
British Library
bl.uk_lddc_bot
allow
bne.es_bot
archiver
bne.es_bot
allow
bnf.fr_bot
archiver
Bibliothèque nationale de France
bnf.fr_bot
allow
CUCH.org
archiver
C U Cyber History
CUCH.org
allow
heritrix
archiver
Internet Archive
heritrix
allow
ia_archiver
archiver
Internet Archive
ia_archiver
allow
ia_archiver-web.archive.org
archiver
Internet Archive
ia_archiver-web.archive.org
allow
IABot
archiver
Wikimedia Foundation
IABot
allow
Internet
archiver
Internet Archive
Internet
allow
Internet Archive
archiver
Internet Archive
archive.org_bot
allow
Internet Archive - Archive-It
archiver
Archive-It
allow
mirrorweb
archiver
MirrorWeb
mirrorweb
allow
netarkivindsamling
archiver
Netarkivet
netarkivindsamling
allow
NewsBank
archiver
NewsBank
allow
Nicecrawler
archiver
NiceCrawler
Nicecrawler
allow
special_archiver
archiver
Internet Archive
special_archiver
allow
Turnitin
archiver
Turnitin
Turnitin
allow
XY-Archive-Compliance-Archiver
archiver
XY Planning Network
XY-Archive-Compliance-Archiver
allow