meta-webindexer
recommended: observemeta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more relevant and accurate, allowing proper citation and linking back to source content.
What it does
meta-webindexer revisits your site on a flexible schedule, often requesting several pages in a burst. Activity usually increases when your content changes or its index refreshes, with quiet intervals between crawl passes.
How to identify it
User-agent contains any of:
meta-webindexer
Example user-agent string:
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)robots.txt
Well-behaved crawlers honor robots.txt. To control meta-webindexer:
Allow full access
User-agent: meta-webindexer
Allow: /Block entirely
User-agent: meta-webindexer
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See meta-webindexer on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes