Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
GeedoProductSearch
search engine
Geedo
GeedoProductSearch
allow
rDNS
GeistHaus-PageFetcher
ai assistant
GeistHaus
GeistHaus-PageFetcher
observe
Gemini-Deep-Research
ai assistant
Gemini-Deep-Research
observe
Genieo
other
Genieo
observe
GenomeCrawlerd
other
Nokia
GenomeCrawlerd
observe
GermCrawler
other
GermCrawler
observe
GetRight
other
GetRight
observe
Ghost Inspector
monitoring
Ghost Inspector
Ghost
allow
gigablast
other
gigablast
observe
Gigabot
search engine
Gigablast
Gigabot
allow
GingerCrawler
other
GingerCrawler
observe
GitHub Copilot Agent
ai coding agent
Microsoft / GitHub
allow
Gluten Free Crawler
other
Gluten
observe
gnam
other
gnam
observe
GnowitNewsbot
other
Gnowit
GnowitNewsbot
observe
Goodreads
ai assistant
Goodreads
Goodreads
observe
Google AdsBot
seo crawler
AdsBot-Google
challenge
Google AdSense
ai assistant
observe
rDNS
Google Adsense: Mediapartners-Google
seo crawler
Mediapartners-Google
challenge
Google API
other
APIs-Google
allow
rDNS
Google AppsViewer
ai assistant
observe
Google Calendar Importer
social media
Google-Calendar-Importer
allow
Google Digital Asset Links
seo crawler
Google Digital Asset Links
challenge
Google Favicon
search engine
allow
Google Feed Fetcher
social media
allow
Google Firebase
seo crawler
Google-Firebase
challenge
Google Image Proxy
social media
GoogleImageProxy
allow
Google Images
search engine
Googlebot-Image
allow
Google Inspection Tool
security
Google Inspection Tool
allow
rDNS
Google Page Renderer
social media
Google-PageRenderer
allow
Google Publisher Center
social media
Google Publisher Center
allow
Google Read Aloud
other
Google-Read-Aloud
allow
Google Safety
security
Google-Safety
allow
rDNS
Google Schema Markup Testing Tool
monitoring
Google-Structured-Data-Testing
allow
Google Scholar
search engine
Google Scholar
allow
Google Site Verification
security
Google-Site-Verification
allow
rDNS
Google StoreBot
seo crawler
Storebot-Google
challenge
rDNS
Google Trust Services (DCV Check)
security
Google-Trust-Services
allow
Google Videos
search engine
Googlebot-Video
allow
Google Web Preview
ai assistant
observe
Google Web Snippet
social media
Google-PageRenderer
allow
Google-Ads-Creatives-Assistant
other
Google-Ads-Creatives-Assistant
observe
Google-adstxt
ai assistant
Google-adstxt
observe
Google-AdWords-Express
seo crawler
Google-AdWords-Express
challenge
Google-Adwords-Instant
seo crawler
Google-Adwords-Instant
challenge
Google-Agent
ai assistant
Google-Agent
observe
rDNS
Google-BusinessLinkVerification
security
Google-BusinessLinkVerifi
allow
Google-Calendar-Importer
other
Google-Calendar-Importer
observe
Google-Certificates-Bridge
other
Google-Certificates-Bridge
observe
Google-CloudVertexBot
ai crawler
Google-CloudVertexBot
observe
rDNS
Google-Display-Ads-Bot
seo crawler
Google-Display-Ads-Bot
challenge
Google-Extended
ai scraper
Google-Extended
observe
rDNS
Google-Flights-Search-Spackle
search engine
Google-Flights-Search-Spackle
allow
Google-Gemini-CLI
ai coding agent
Google-Gemini-CLI
observe
Google-InspectionTool
security
Google-InspectionTool
allow
Google-NotebookLM
ai crawler
Google-NotebookLM
allow
Google-PageRenderer
ai assistant
Google-PageRenderer
observe
Google-PhysicalWeb
other
Google-PhysicalWeb
observe
Google-Read-Aloud
ai assistant
Google-Read-Aloud
observe
Google-Structured-Data-Testing
seo crawler
Google-Structured-Data-Testing
challenge