Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
Google Images
search engine
allow
Google Inspection Tool
security
Google
allow
Google Page Renderer
social media
Google-PageRenderer
allow
Google Publisher Center
social media
Google
allow
Google Read Aloud
other
Google-Read-Aloud
allow
Google Safety
security
Google-Safety
allow
Google Schema Markup Testing Tool
monitoring
allow
Google Scholar
search engine
Google
allow
Google Site Verification
security
Google-Site-Verification
allow
Google StoreBot
seo crawler
Storebot-Google
challenge
Google Trust Services (DCV Check)
security
Google-Trust-Services
allow
Google Videos
search engine
allow
Google Web Snippet
social media
allow
Google-Ads-Creatives-Assistant
other
Google-Ads-Creatives-Assistant
observe
Google-adstxt
ai assistant
Google-adstxt
observe
Google-AdWords-Express
seo crawler
Google-AdWords-Express
challenge
Google-Adwords-Instant
seo crawler
Google-Adwords-Instant
challenge
Google-Agent
ai assistant
Google-Agent
observe
Google-BusinessLinkVerification
security
allow
Google-Certificates-Bridge
other
Google-Certificates-Bridge
observe
Google-CloudVertexBot
ai crawler
Google-CloudVertexBot
observe
Google-Display-Ads-Bot
seo crawler
Google-Display-Ads-Bot
challenge
Google-Extended
ai scraper
Google-Extended
observe
Google-Flights-Search-Spackle
search engine
allow
Google-Gemini-CLI
ai coding agent
Google-Gemini-CLI
observe
Google-NotebookLM
ai crawler
Google-NotebookLM
allow
Google-PhysicalWeb
other
Google-PhysicalWeb
observe
Google-Structured-Data-Testing
seo crawler
Google-Structured-Data-Testing
challenge
Google-Structured-Data-Testing-Tool
ai coding agent
Google-Structured-Data-Testing-Tool
observe
Google-TravelAds-Live
seo crawler
challenge
google-xrawler
search engine
google-xrawler
allow
GoogleAgent-Mariner
browser automation
Google DeepMind
GoogleAgent-Mariner
observe
GoogleAgent-URLContext
ai assistant
GoogleAgent-URLContext
observe
GoogleAssociationService
ai coding agent
GoogleAssociationService
observe
GoogleBot
search engine
Googlebot
allow
Googlebot-IA
search engine
Googlebot-IA
allow
Googlebot-Image
search engine
Googlebot-Image
allow
Googlebot-Mobile
search engine
Googlebot-Mobile
allow
Googlebot-News
search engine
Googlebot-News
allow
Googlebot-Video
search engine
Googlebot-Video
allow
GoogleDocs
ai assistant
GoogleDocs
observe
GoogleOther
ai crawler
GoogleOther
observe
GoogleProducer
ai assistant
GoogleProducer
observe
GoParserBot
other
GoParserBot
observe
GoPay
other
GoPay.cz
allow
Gowikibot
other
Gowikibot
observe
GPTBot
ai crawler
OpenAI
GPTBot
observe
grafana
ai coding agent
Grafana Labs
grafana
observe
Grafana Synthetic Monitoring
monitoring
Grafana Labs
allow
Grafana's
ai coding agent
Grafana Labs
Grafana's
observe
Grafana's Synthetic Monitoring
monitoring
Grafana Labs
allow
Grammarly
ai assistant
Grammarly
Grammarly
observe
Grapeshot
seo crawler
Grapeshot
grapeshot
challenge
GrapeshotCrawler
ai assistant
Grapeshot
GrapeshotCrawler
observe
Greppr
search engine
Greppr
Greppr
allow
Grobbot
other
Grobbot
observe
GroovinaAdsbot
ai assistant
Groovinads
GroovinaAdsbot
observe
Groovinads
seo crawler
Groovinads
challenge
GroupHigh
other
GroupHigh
observe
grub-client
other
grub-client
observe