Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

fr-crawler other

fr-crawler

observe

FreeWebMonitoring SiteChecker other

FreshpingBot ai coding agent

Freshworks

FreshpingBot

observe

FreshRSS ai assistant

FreshRSS

FreshRSS

observe

Friendica ai assistant

Friendica

G2 Web Services other

GeedoShopProductFinder other

GeedoShopProductFinder

observe rDNS

GeistHaus-PageFetcher ai assistant

GeistHaus

GeistHaus-PageFetcher

observe

Gemini-Deep-Research ai assistant

Google

Gemini-Deep-Research

observe

GeneralCrawlBot other

Nokia

getdomaindata ai assistant

getdomaindata.com

GhostExplore ai assistant

Ghost

GlobRadarBot ai assistant

GlobRadar

GlobRadarBot

observe

Gluten Free Crawler other

Gnowit

Golgul

Goodreads ai assistant

Goodreads

Goodreads

observe

Google AdSense ai assistant

Google

observe rDNS

Google AppsViewer ai assistant

Google

observe

Google Web Preview ai assistant

Google

observe

Google-Ads-Creatives-Assistant other

Google-Ads-Creatives-Assistant

observe

Google-adstxt ai assistant

Google

Google-adstxt

observe

Google-Agent ai assistant

Google

Google-Agent

observe rDNS

Google-Certificates-Bridge other

Google-Certificates-Bridge

observe

Google-CloudVertexBot ai crawler

Google

Google-CloudVertexBot

observe rDNS

Google-Extended ai scraper

Google

Google-Extended

observe rDNS

Google-Gemini-CLI ai coding agent

Google

Google-Gemini-CLI

observe

Google-PhysicalWeb other

Google-PhysicalWeb

observe

Google-Structured-Data-Testing-Tool ai coding agent

Google

Google-Structured-Data-Testing-Tool

observe

GoogleAgent-Mariner browser automation

Google DeepMind

GoogleAgent-Mariner

observe

GoogleAgent-URLContext ai assistant

Google

GoogleAgent-URLContext

observe

GoogleAssociationService ai coding agent

Google

GoogleAssociationService

observe

GoogleDocs ai assistant

Google

GoogleDocs

observe

GoogleMessages ai assistant

Google

GoogleMessages

observe rDNS

GoogleOther ai crawler

Google

GoogleOther

observe rDNS

GoogleProducer ai assistant

Google

GovAPIEnricher ai assistant

Mega Impacts

OpenAI

grafana ai coding agent

Grafana Labs

grafana

observe

Grafana's Synthetic Monitoring ai coding agent

Grafana Labs

Grafana&

observe

Grammarly ai assistant

Grammarly

Grammarly

observe

GrapeshotCrawler ai assistant

Grapeshot

GrapeshotCrawler

observe

← Prev 9 / 24 Next →