Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

2,114 results

FirmCredBot ai assistant

Firm Cred

FirmlyAI Bot ai assistant

Firmly

FishBot ai crawler

FishBot

FIT Feeds Fetcher social media

follow.it

Flaming AttackBot other

Flamingo_SearchEngine other

Flamingo_SearchEngine

Flashy-PHP/1.0.0 other

Flashy

FleebsBot other

Fleebs

FlipboardProxy social media

Flipboard

FlipboardProxy

FlipboardRSS social media

Flipboard

FlockWP ai assistant

FlockWP

FlowIQLabsBot ai assistant

FlowIQ Labs

FlowIQLabsBot

FlyingPress seo crawler

FlyingWeb

Flyriverbot other

Foregenix security

Foregenix

Foregenix ThreatView/WebScan monitoring

Foregenix Limited

ForestEngine ai assistant

Forest

foureff ai assistant

BITS - Benedikt IT Services e.U.

fr-crawler other

Freespoke search engine

Freespoke

crawler.freespoke.com

FreespokeCrawler search engine

Freespoke

FreespokeCrawler

FreeWebMonitoring SiteChecker other

FreeWebMonitoring

Freshping other

Freshworks

FreshpingBot ai coding agent

Freshworks

FreshRSS ai assistant

FreshRSS

Friendica ai assistant

Friendica

FrogleBot search engine

Frogle

FullFact Ingest social media

FullFact

FullStory monitoring

Full Story

Funnelback search engine

Squiz - FunnelBack

FusionBot other

fyndbot search engine

Fynd

FyndSearchEngine-Crawler search engine

Fynd

FyndSearchEngine-Crawler

FyndSearchEngine-ReCrawler search engine

FyndSearchEngine-ReCrawler

g00g1e.net other

G2 Web Services other

g2reader-bot other

GarlikCrawler other

GarlikCrawler

Gatus monitoring

Gatus

GeedoProductSearch search engine

Geedo

GeedoProductSearch

GeedoShopProductFinder other

GeedoShopProductFinder

GeistHaus-PageFetcher ai assistant

GeistHaus

GeistHaus-PageFetcher

Gemini-Deep-Research ai assistant

Google

Gemini-Deep-Research

GeneralCrawlBot other

GeneralCrawlBot

GenomeCrawlerd other

Nokia

GenomeCrawlerd

Gensparkbot other

GermCrawler other

getdomaindata ai assistant

getdomaindata.com

getdomaindata

Ghost Inspector monitoring

Ghost Inspector

GhostExplore ai assistant

Ghost

GidoraBot search engine

Gidora

← Prev 12 / 36 Next →