Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
812 results · Other
FedReporter
other
Fed Reporter
FedReporter
observe
FeedDiscovery
other
FeedDiscovery
observe
Feedsearch
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
FeedValidator
other
FeedValidator
observe
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe
Fever
other
Fever
observe
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
fiperbot
observe
Flaming
other
Flaming
observe
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
Flashy-PHP/1.0.0
other
Flashy
allow
FleebsBot
other
Fleebs
FleebsBot
observe
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
fr-crawler
other
fr-crawler
observe
FreeWebMonitoring
other
FreeWebMonitoring
observe
Freshbot
other
Freshbot
observe
Freshping
other
Freshworks
Freshping
allow
fuelbot
other
fuelbot
observe
Fyrebot
other
Fyrebot
observe
g00g1e.net
other
g00g1e.net
observe
G2
other
G2
observe
g2reader-bot
other
g2reader-bot
observe
Gaisbot
other
Gaisbot
observe
GarlikCrawler
other
GarlikCrawler
observe
Genieo
other
Genieo
observe
GenomeCrawlerd
other
Nokia
GenomeCrawlerd
observe
GermCrawler
other
GermCrawler
observe
GetRight
other
GetRight
observe
gigablast
other
gigablast
observe
GingerCrawler
other
GingerCrawler
observe
Gluten
other
Gluten
observe
gnam
other
gnam
observe
GnowitNewsbot
other
GnowitNewsbot
observe
Google API
other
Google
allow
Google Read Aloud
other
Google-Read-Aloud
allow
Google-Ads-Creatives-Assistant
other
Google-Ads-Creatives-Assistant
observe
Google-Certificates-Bridge
other
Google-Certificates-Bridge
observe
Google-PhysicalWeb
other
Google-PhysicalWeb
observe
GoParserBot
other
GoParserBot
observe
GoPay
other
GoPay.cz
allow
Gowikibot
other
Gowikibot
observe
Grobbot
other
Grobbot
observe
GroupHigh
other
GroupHigh
observe
grub-client
other
grub-client
observe
grub.org
other
grub.org
observe
gsa-crawler
other
gsa-crawler
observe
gslfbot
other
gslfbot
observe
Gwene
other
Gwene
observe
HaloBot
other
Haloscan
HaloBot
observe
HanaleiBot
other
HanaleiBot
observe
Harvest
other
Harvest
observe
HawaiiBot
other
HawaiiBot
observe
HeadlessChrome
other
HeadlessChrome
observe