Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
Flashy-PHP/1.0.0
other
Flashy
allow
FleebsBot
other
Fleebs
FleebsBot
observe
FlipboardProxy
social media
FlipboardProxy
allow
FlipboardRSS
social media
FlipboardRSS
allow
FlockWP
ai assistant
FlockWP
observe
signed
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
Foregenix
ai coding agent
Foregenix
Foregenix
observe
Foregenix ThreatView/WebScan
monitoring
Foregenix Limited
allow
foureff
ai assistant
BITS - Benedikt IT Services e.U.
observe
signed
fr-crawler
other
fr-crawler
observe
Freespoke
search engine
Freespoke
Freespoke
allow
FreespokeCrawler
search engine
Freespoke
FreespokeCrawler
allow
FreeWebMonitoring
other
FreeWebMonitoring
observe
Freshbot
other
Freshbot
observe
Freshping
other
Freshworks
Freshping
allow
FreshpingBot
ai coding agent
Freshworks
FreshpingBot
observe
FreshRSS
ai assistant
FreshRSS
FreshRSS
observe
Friendica
ai assistant
Friendica
Friendica
observe
fuelbot
other
fuelbot
observe
FullFact Ingest
social media
FullFact
allow
FullStory
monitoring
Full Story
FullStoryBot
allow
Funnelback
search engine
Squiz - FunnelBack
Funnelback
allow
fyndbot
search engine
Fynd
fyndbot
allow
FyndSearchEngine-Crawler
search engine
Fynd
FyndSearchEngine-Crawler
allow
FyndSearchEngine-ReCrawler
search engine
FyndSearchEngine-ReCrawler
allow
Fyrebot
other
Fyrebot
observe
g00g1e.net
other
g00g1e.net
observe
G2
other
G2
observe
g2reader-bot
other
g2reader-bot
observe
Gaisbot
other
Gaisbot
observe
GarlikCrawler
other
GarlikCrawler
observe
Gatus
monitoring
Gatus
allow
GeedoProductSearch
search engine
Geedo
GeedoProductSearch
allow
GeistHaus-PageFetcher
ai assistant
GeistHaus
GeistHaus-PageFetcher
observe
Gemini-Deep-Research
ai assistant
Gemini-Deep-Research
observe
Genieo
other
Genieo
observe
GenomeCrawlerd
other
Nokia
GenomeCrawlerd
observe
GermCrawler
other
GermCrawler
observe
GetRight
other
GetRight
observe
Ghost Inspector
monitoring
Ghost Inspector
Ghost
allow
gigablast
other
gigablast
observe
Gigabot
search engine
Gigablast
Gigabot
allow
GingerCrawler
other
GingerCrawler
observe
GitHub Copilot Agent
ai coding agent
Microsoft / GitHub
allow
Gluten
other
Gluten
observe
gnam
other
gnam
observe
GnowitNewsbot
other
GnowitNewsbot
observe
Goodreads
ai assistant
Goodreads
Goodreads
observe
Google AdsBot
seo crawler
challenge
Google Adsense: Mediapartners-Google
seo crawler
challenge
Google API
other
Google
allow
Google Calendar Importer
social media
Google-Calendar-Importer
allow
Google Digital Asset Links
seo crawler
Google
challenge
Google Feed Fetcher
social media
allow
Google Firebase
seo crawler
challenge
Google Image Proxy
social media
GoogleImageProxy
allow