Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Feedfetcher-Google
search engine
Feedfetcher-Google
allow
rDNS
FeedFlow
ai assistant
FeedFlow
FeedFlow
observe
Feedly
social media
Feedly
Feedly
allow
FeedOtter
social media
FeedOtter
FeedOtter
allow
Feedsearch Bot
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
Feedspot
ai assistant
Feedspot
Feedspot
observe
Feedspot SSL asset Proxy
other
observe
FeedValidator
other
FeedValidator
observe
FeedWind Crawler
social media
Mikle
Feedwind
allow
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe
Fever
other
Fever
observe
FindFiles.net
search engine
FindFiles.net
FindFiles.net
allow
FindITAnswersbot
search engine
IT Influentials
FindITAnswersbot
allow
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
fiperbot
observe
FirecrawlAgent
ai crawler
Mendable / Firecrawl
FirecrawlAgent
observe
FirmlyAI Bot
ai assistant
Firmly
firmlyai-bot
observe
FishBot
ai crawler
FishBot
FishBot
observe
FIT Feeds Fetcher
social media
follow.it
follow.it
allow
Flaming AttackBot
other
Flaming
observe
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
Flashy-PHP/1.0.0
other
Flashy
Flashy-PHP
allow
FleebsBot
other
Fleebs
FleebsBot
observe
FlipboardProxy
social media
FlipboardProxy
allow
FlipboardRSS
social media
FlipboardRSS
allow
FlockWP
ai assistant
FlockWP
FlockWP
observe
signed
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
Foregenix
security
Foregenix
Foregenix
allow
Foregenix ThreatView/WebScan
monitoring
Foregenix Limited
Foregenix
allow
foureff
ai assistant
BITS - Benedikt IT Services e.U.
observe
signed
fr-crawler
other
fr-crawler
observe
Freespoke
search engine
Freespoke
crawler.freespoke.com
allow
FreespokeCrawler
search engine
Freespoke
FreespokeCrawler
allow
FreeWebMonitoring SiteChecker
other
FreeWebMonitoring
observe
Freshbot
other
Freshbot
observe
Freshping
other
Freshworks
Freshping
allow
FreshpingBot
ai coding agent
Freshworks
FreshpingBot
observe
FreshRSS
ai assistant
FreshRSS
FreshRSS
observe
Friendica
ai assistant
Friendica
Friendica
observe
fuelbot
other
fuelbot
observe
FullFact Ingest
social media
FullFact
FullFact
allow
FullStory
monitoring
Full Story
FullStoryBot
allow
Funnelback
search engine
Squiz - FunnelBack
Funnelback
allow
fyndbot
search engine
Fynd
fyndbot
allow
FyndSearchEngine-Crawler
search engine
Fynd
FyndSearchEngine-Crawler
allow
FyndSearchEngine-ReCrawler
search engine
FyndSearchEngine-ReCrawler
allow
Fyrebot
other
Fyrebot
observe
g00g1e.net
other
g00g1e.net
observe
G2 Web Services
other
observe
g2reader-bot
other
g2reader-bot
observe
Gaisbot
other
Gaisbot
observe
GarlikCrawler
other
GarlikCrawler
observe
Gatus
monitoring
Gatus
Gatus
allow