Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Integromat
other
Make
Integromat
allow
integromedb
other
integromedb
observe
intelium_bot
other
intelium_bot
observe
intelx.io_bot
search engine
Intelligence X
intelx.io_bot
allow
InteractiveAdvertisingBureau
ai assistant
OneTag
InteractiveAdvertisingBureau
observe
InterfaxScanBot
other
InterfaxScanBot
observe
Internet Archive
archiver
Internet Archive
archive.org_bot
allow
Internet Archive - Archive-It
archiver
Archive-It
Archive-It
allow
InternetArchiveBot
monitoring
Internet Archive
InternetArchiveBot
allow
InternetMeasurement
security
Driftnet
InternetMeasurement
allow
IonCrawl
ai assistant
IONOS
IonCrawl
observe
ip-web-crawler.com
other
ip-web-crawler.com
observe
ips-agent
other
ips-agent
observe
IRLbot
other
IRLbot
observe
Iron33
other
Iron33
observe
IsDownBot
monitoring
IsDownBot
IsDownBot
allow
iskanie
other
iskanie
observe
IsraBot
other
IsraBot
observe
ISSCyberRiskCrawler
security
ISS
ISSCyberRiskCrawler
allow
istellabot
other
istellabot
observe
it2media-domain-crawler
other
it2media-domain-crawler
observe
Jagged Pixel UptimeBot
monitoring
Jagged Pixel Inc.
www.getuptime.co
allow
Jambot
search engine
JamBot
Jambot
allow
James BOT
seo crawler
cognitiveSEO
James
challenge
JamesBOT
other
JamesBOT
observe
Jamie's Spider
other
observe
JenkersBot
other
JenkersBot
observe
JennyBot
other
JennyBot
observe
Jetbot
other
Jetbot
observe
jetmon
ai coding agent
Automattic
jetmon
observe
Jetpack
monitoring
Automattic
Jetpack
allow
Jetty
other
Jetty
observe
JikeSpider
other
JikeSpider
observe
JobboerseBot
other
JobboerseBot
observe
JobHuntBot
other
JobHuntBot
observe
Jobiglo-Bot
search engine
Jobiglo
Jobiglo-Bot
allow
jobswithgptcom-bot
search engine
JobsWithGPT
jobswithgptcom-bot
allow
Jooblebot
search engine
Jooble
Jooblebot
allow
jpg-newsbot
other
jpg-newsbot
observe
Jugendschutzprogramm-Crawler
ai assistant
JusProg
Jugendschutzprogramm-Crawler
observe
Jumio
security
Coinbase
Jumio
allow
jyxobot
other
jyxobot
observe
k2spider
other
k2spider
observe
K7MLWCBot
security
K7 Computing
K7MLWCBot
allow
Kagi Bot
search engine
Kagi
allow
kagi-fetcher
ai assistant
Kagi
kagi-fetcher
observe
Kagibot
search engine
Kagi
Kagibot
allow
rDNS
kaikki.org-digital-archive
other
kaikki.org-digital-archive
observe
KakaoTalk Scrap
social media
Kakao
kakaotalk-scrap
allow
Kangaroo Bot
ai scraper
Kangaroo LLM
Kangaroo
observe
KargoBot
ai assistant
Kargo
KargoBot
observe
KargoBot-Artemis
monitoring
Kargo
KargoBot-Artemis
allow
KargoBot-Artemis-Mobile
ai assistant
Kargo
KargoBot-Artemis-Mobile
observe
KauaiBot
other
KauaiBot
observe
kb.dk_bot
ai scraper
Netarkivet
kb.dk_bot
observe
kbcrawl
other
kbcrawl
observe
Kemvibot
other
Kemvibot
observe
Kenjin Spider
other
Kenjin
observe
Kernel
ai assistant
Kernel
kernel
observe
signed
Keydrop.io
other
Keydrop.io
observe