Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

2,114 results

gigablast other

gigablast

observe

Gigabot search engine

Gigablast

GitHub Copilot Agent ai coding agent

Microsoft / GitHub

allow

GlobRadarBot ai assistant

GlobRadar

GlobRadarBot

observe

Gluten Free Crawler other

Gnowit

Golgul

Goodreads ai assistant

Goodreads

Goodreads

observe

Google AdsBot seo crawler

Google

AdsBot-Google

allow

Google AdSense ai assistant

Google

observe rDNS

Google Adsense: Mediapartners-Google seo crawler

Google

Mediapartners-Google

Google

Google AppsViewer ai assistant

Google

observe

Google Calendar Importer social media

Google

Google-Calendar-Importer

allow

Google Digital Asset Links seo crawler

Google

Google Digital Asset Links

challenge

Google Favicon search engine

Google

allow

Google Feed Fetcher social media

Google

allow

Google Firebase seo crawler

Google

Google-Firebase

challenge

Google Image Proxy social media

Google

GoogleImageProxy

allow

Google Images search engine

Google

Googlebot-Image

allow

Google Inspection Tool security

Google

Google Inspection Tool

allow rDNS

Google Page Renderer social media

Google

Google-PageRenderer

allow

Google Publisher Center social media

Google

Google Publisher Center

allow

Google Read Aloud other

Google

Google-Read-Aloud

allow

Google Safety security

Google

Google-Safety

allow rDNS

Google Schema Markup Testing Tool monitoring

Google

Google-Structured-Data-Testing

allow

Google Scholar search engine

Google

Google Scholar

allow

Google Site Verification security

Google

Google-Site-Verification

allow rDNS

Google StoreBot seo crawler

Google

Storebot-Google

allow rDNS

Google Trust Services (DCV Check) security

Google

Google-Trust-Services

allow

Google Videos search engine

Google

Googlebot-Video

allow

Google Web Preview ai assistant

Google

observe

Google Web Snippet social media

Google

Google-PageRenderer

allow

Google-Ads-Creatives-Assistant other

Google-Ads-Creatives-Assistant

observe

Google-adstxt ai assistant

Google

Google-adstxt

observe

Google-AdWords-Express seo crawler

Google

Google-AdWords-Express

challenge

Google-Adwords-Instant seo crawler

Google

Google-Adwords-Instant

challenge

Google-Agent ai assistant

Google

Google-Agent

observe rDNS

Google-BusinessLinkVerification security

Google

Google-BusinessLinkVerifi

allow

Google-Certificates-Bridge other

Google-Certificates-Bridge

observe

Google-CloudVertexBot ai crawler

Google

Google-CloudVertexBot

observe rDNS

Google-Display-Ads-Bot seo crawler

Google

Google-Display-Ads-Bot

challenge

Google-Extended ai scraper

Google

Google-Extended

observe rDNS

Google-Flights-Search-Spackle search engine

Google

Google-Flights-Search-Spackle

allow

Google-Gemini-CLI ai coding agent

Google

Google-Gemini-CLI

observe

Google-NotebookLM ai crawler

Google

Google-NotebookLM

allow

Google-PhysicalWeb other

Google-PhysicalWeb

observe

Google-Structured-Data-Testing seo crawler

Google

Google-Structured-Data-Testing

challenge

Google-Structured-Data-Testing-Tool ai coding agent

Google

Google-Structured-Data-Testing-Tool

observe

Google-TravelAds-Live seo crawler

Google

Google-TravelAds-Live

challenge

google-xrawler search engine

Google

google-xrawler

allow

GoogleAgent-Mariner browser automation

Google DeepMind

GoogleAgent-Mariner

observe

GoogleAgent-URLContext ai assistant

Google

GoogleAgent-URLContext

observe

GoogleAssociationService ai coding agent

Google

GoogleAssociationService

observe

GoogleBot search engine

Google

Googlebot

allow rDNS

Googlebot-IA search engine

Google

Googlebot-IA

allow rDNS

Googlebot-Image search engine

Google

Googlebot-Image

allow rDNS

Googlebot-Mobile search engine

Google

Googlebot-Mobile

allow rDNS

← Prev 13 / 36 Next →