Skip to main content
Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

497 results

elmah.io Uptime Monitoring monitoring

elmah.io

elmahio-uptimebot
allow
Embedly social media

Embedly

Embedly
allow
eRepublik.tools monitoring

Sebastian Foth - Software Solutions

eRepublik.tools
allow
EtherdeckBot search engine

Etherdeck

EtherdeckBot
allow
EvoUptimeBot monitoring

Evo Agency Ltd.

EvoUptimeBot
allow
Exodus other

Exodus

ExodusMovement
allow
Expanse security

Palo Alto Networks

Expanse
allow
FacebookExternalHit social media

Meta

facebookexternalhit
allow
FastDAST security

Black Duck

FastDAST
allow
Fastmail Bot social media

Fastmail

Fastmail Bot
allow
FDL Stats Bots monitoring

FTW Entertainment LLC

FDL
allow
Feedbin social media

Feedbin

Feedbin
allow
Feeder social media

Really Simple AB

Feeder
allow
Feedfetcher-Google search engine

Google

Feedfetcher-Google
allow rDNS
Feedly social media

Feedly

Feedly
allow
FeedOtter social media

FeedOtter

FeedOtter
allow
FeedWind Crawler social media

Mikle

Feedwind
allow
FindFiles.net search engine

FindFiles.net

FindFiles.net
allow
FindiseBot search engine

Findise

FindiseBot
allow
FindITAnswersbot search engine

IT Influentials

FindITAnswersbot
allow
FIT Feeds Fetcher social media

follow.it

follow.it
allow
Flashy-PHP/1.0.0 other

Flashy

Flashy-PHP
allow
FlipboardProxy social media

Flipboard

FlipboardProxy
allow
FlipboardRSS social media

Flipboard

FlipboardRSS
allow
Foregenix security

Foregenix

Foregenix
allow
Foregenix ThreatView/WebScan monitoring

Foregenix Limited

Foregenix
allow
Freespoke search engine

Freespoke

crawler.freespoke.com
allow
FreespokeCrawler search engine

Freespoke

FreespokeCrawler
allow
Freshping other

Freshworks

Freshping
allow
FrogleBot search engine

Frogle

FrogleBot
allow
FullFact Ingest social media

FullFact

FullFact
allow
FullStory monitoring

Full Story

FullStoryBot
allow
Funnelback search engine

Squiz - FunnelBack

Funnelback
allow
fyndbot search engine

Fynd

fyndbot
allow
FyndSearchEngine-Crawler search engine

Fynd

FyndSearchEngine-Crawler
allow
FyndSearchEngine-ReCrawler search engine
FyndSearchEngine-ReCrawler
allow
Gatus monitoring

Gatus

Gatus
allow
GeedoProductSearch search engine

Geedo

GeedoProductSearch
allow rDNS
Ghost Inspector monitoring

Ghost Inspector

Ghost
allow
GidoraBot search engine

Gidora

GidoraBot
allow
Gigabot search engine

Gigablast

Gigabot
allow
GitHub Copilot Agent ai coding agent

Microsoft / GitHub

allow
Google AdsBot seo crawler

Google

AdsBot-Google
allow
Google API other

Google

APIs-Google
allow rDNS
Google Calendar Importer social media

Google

Google-Calendar-Importer
allow
Google Favicon search engine

Google

allow
Google Feed Fetcher social media

Google

allow
Google Image Proxy social media

Google

GoogleImageProxy
allow
Google Images search engine

Google

Googlebot-Image
allow
Google Inspection Tool security

Google

Google Inspection Tool
allow rDNS
Google Page Renderer social media

Google

Google-PageRenderer
allow
Google Publisher Center social media

Google

Google Publisher Center
allow
Google Read Aloud other

Google

Google-Read-Aloud
allow
Google Safety security

Google

Google-Safety
allow rDNS
Google Schema Markup Testing Tool monitoring

Google

Google-Structured-Data-Testing
allow
Google Scholar search engine

Google

Google Scholar
allow
Google Site Verification security

Google

Google-Site-Verification
allow rDNS
Google StoreBot seo crawler

Google

Storebot-Google
allow rDNS
Google Trust Services (DCV Check) security

Google

Google-Trust-Services
allow
Google Videos search engine

Google

Googlebot-Video
allow
← Prev 3 / 9 Next →