CloudflareBrowserRenderingCrawler
recommended: observeCloudflareBrowserRenderingCrawler is a web crawler operated by Cloudflare that returns rendered website content for research, monitoring, and AI data workflows.
What it does
CloudflareBrowserRenderingCrawler supports several kinds of automated activity, so its traffic pattern depends on the task. It may perform broad crawls to collect content or build an index, make targeted requests for specific pages, or shift between these behaviors as demand changes.
How to identify it
User-agent contains any of:
CloudflareBrowserRenderingCrawler
robots.txt
Well-behaved crawlers honor robots.txt. To control CloudflareBrowserRenderingCrawler:
Allow full access
User-agent: CloudflareBrowserRenderingCrawler
Allow: /Block entirely
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See CloudflareBrowserRenderingCrawler on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes