Terra

Consider your priorities. Terra Cotta crawls websites on behalf of its customers to supply data for AI training, search, and retrieval-augmented generation. Your content may be redistributed to many downstream AI applications through a single provider. You may want to block it if you're concerned about how your content is being used across those systems, or allow it if you value the discoverability and reach it can provide.

If you want to, you can block or limit Terra Cotta's access by configuring user agent token rules in your robots.txt file. The best way to do this is using Automatic Robots.txt, which update automatically as new agents are discovered. While the vast majority of agents operated by reputable companies honor these robots.txt directives, bad actors may choose to ignore them entirely. In that case, you'll need to implement alternative blocking methods such as firewall rules or server-level restrictions. You can verify whether Terra Cotta is respecting your rules by setting up Agent Analytics to monitor its visits to your website.

Blocking AI data providers has no direct impact on traditional SEO rankings since they don't control search engine indexing. However, these services feed content into AI search engines, RAG pipelines, and conversational AI platforms. Blocking them could reduce your content's representation across multiple AI-powered discovery channels simultaneously, since a single provider may supply data to many downstream applications.

AI data providers typically crawl publicly accessible web content to build their indexes and fulfill API requests. Some providers operate large-scale proxy networks and may attempt to access content aggressively or bypass rate limits. The scope depends on what their customers request and the provider's own indexing priorities. Most focus on public content, but their scale and the diversity of downstream use cases mean your content could be accessed more broadly than with a single-purpose crawler.

Setting up Agent Analytics will give you realtime visibility into Terra Cotta visiting your website, along with hundreds of other AI agents, crawlers, and scrapers. This will also let you measure human traffic to your website coming from AI search and chat LLM platforms like ChatGPT, Perplexity, and Gemini.

Terra Cotta crawled your site to fulfill data requests from its customers or to build and maintain its own web index. Your site was likely identified as containing content relevant to AI training datasets, search indexes, or retrieval-augmented generation pipelines. The crawl may have been triggered by a specific customer API request or as part of the provider's broader web indexing efforts.

Agent Analytics authenticates agent visits from many agents, letting you know whether each one was actually from that agent, or spoofed by a bad actor. This helps you identify suspicious traffic patterns and make informed decisions about blocking or allowing specific user agents.