Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
Fever
other
Fever
observe
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
fiperbot
observe
FirecrawlAgent
ai crawler
Mendable / Firecrawl
FirecrawlAgent
observe
FirmlyAI Bot
ai assistant
Firmly
firmlyai-bot
observe
FishBot
ai crawler
FishBot
FishBot
observe
Flaming AttackBot
other
Flaming
observe
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
FleebsBot
other
Fleebs
FleebsBot
observe
FlockWP
ai assistant
FlockWP
FlockWP
observe
signed
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
foureff
ai assistant
BITS - Benedikt IT Services e.U.
observe
signed
fr-crawler
other
fr-crawler
observe
FreeWebMonitoring SiteChecker
other
FreeWebMonitoring
observe
Freshbot
other
Freshbot
observe
FreshpingBot
ai coding agent
Freshworks
FreshpingBot
observe
FreshRSS
ai assistant
FreshRSS
FreshRSS
observe
Friendica
ai assistant
Friendica
Friendica
observe
fuelbot
other
fuelbot
observe
Fyrebot
other
Fyrebot
observe
g00g1e.net
other
g00g1e.net
observe
G2 Web Services
other
observe
g2reader-bot
other
g2reader-bot
observe
Gaisbot
other
Gaisbot
observe
GarlikCrawler
other
GarlikCrawler
observe
GeistHaus-PageFetcher
ai assistant
GeistHaus
GeistHaus-PageFetcher
observe
Gemini-Deep-Research
ai assistant
Gemini-Deep-Research
observe
Genieo
other
Genieo
observe
GenomeCrawlerd
other
Nokia
GenomeCrawlerd
observe
GermCrawler
other
GermCrawler
observe
GetRight
other
GetRight
observe
gigablast
other
gigablast
observe
GingerCrawler
other
GingerCrawler
observe
Gluten Free Crawler
other
Gluten
observe
gnam
other
gnam
observe
GnowitNewsbot
other
Gnowit
GnowitNewsbot
observe
Goodreads
ai assistant
Goodreads
Goodreads
observe
Google AdSense
ai assistant
observe
rDNS
Google AppsViewer
ai assistant
observe
Google Web Preview
ai assistant
observe
Google-Ads-Creatives-Assistant
other
Google-Ads-Creatives-Assistant
observe
Google-adstxt
ai assistant
Google-adstxt
observe
Google-Agent
ai assistant
Google-Agent
observe
rDNS
Google-Calendar-Importer
other
Google-Calendar-Importer
observe
Google-Certificates-Bridge
other
Google-Certificates-Bridge
observe
Google-CloudVertexBot
ai crawler
Google-CloudVertexBot
observe
rDNS
Google-Extended
ai scraper
Google-Extended
observe
rDNS
Google-Gemini-CLI
ai coding agent
Google-Gemini-CLI
observe
Google-PageRenderer
ai assistant
Google-PageRenderer
observe
Google-PhysicalWeb
other
Google-PhysicalWeb
observe
Google-Read-Aloud
ai assistant
Google-Read-Aloud
observe
Google-Structured-Data-Testing-Tool
ai coding agent
Google-Structured-Data-Testing-Tool
observe
GoogleAgent-Mariner
browser automation
Google DeepMind
GoogleAgent-Mariner
observe
GoogleAgent-URLContext
ai assistant
GoogleAgent-URLContext
observe
GoogleAssociationService
ai coding agent
GoogleAssociationService
observe