AIWebIndex
recommended: observeAIWebIndex is a web crawler operated by Lyrenth that builds an AI-readable index of web content for AI systems.
What it does
AIWebIndex crawls systematically and at volume, because it feeds an index that many AI customers query. Expect recurring visits that cover large parts of your site rather than single pages. Bursts often trace back to a customer request on its end rather than anything on yours.
How to identify it
User-agent contains any of:
AIWebIndex
robots.txt
Well-behaved crawlers honor robots.txt. To control AIWebIndex:
Allow full access
User-agent: AIWebIndex
Allow: /Block entirely
User-agent: AIWebIndex
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See AIWebIndex on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes