KimiCrawler
recommended: observeKimiCrawler is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.
What it does
KimiCrawler crawls websites to build AI training datasets, and its operator decides which sites, how often, and how deep it goes. Expect broad sweeps that fetch far more pages per visit than a search crawler would, on an unpredictable schedule. Volume can run heavy while a collection pass is underway, then stop entirely.
How to identify it
User-agent contains any of:
KimiCrawler
robots.txt
Well-behaved crawlers honor robots.txt. To control KimiCrawler:
Allow full access
User-agent: KimiCrawler
Allow: /Block entirely
User-agent: KimiCrawler
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See KimiCrawler on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes