Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe
ExtractorPro
other
ExtractorPro
observe
Eyeotabot
other
Eyeotabot
observe
EZID
other
EZID
observe
EzLynx
ai assistant
Ezoic
EzLynx
observe
EzoicBot
seo crawler
Ezoic Inc
EzoicBot
challenge
EzoicBot-Nicheiq
ai assistant
Ezoic
EzoicBot-Nicheiq
observe
Ezooms
other
Ezooms
observe
FAB-Scraper
other
FAB-Scraper
observe
FacebookBot
ai scraper
Meta
FacebookBot
observe
FacebookExternalHit
social media
Meta
facebook
allow
Facebot
other
Facebot
observe
Factset_spyderbot
ai crawler
Factset
factset_spyderbot
observe
FairAd
other
FairAd
observe
FAST
other
FAST
observe
FAST-WebCrawler
other
FAST-WebCrawler
observe
FastDAST
security
Black Duck
FastDAST
allow
Fastmail Bot
social media
Fastmail
Fastmail
allow
FastmailUA
ai assistant
Fastmail
FastmailUA
observe
FDL Stats Bots
monitoring
FTW Entertainment LLC
FDL
allow
Fedicabot
seo crawler
Fedica
Fedicabot
challenge
FediDB
other
FediDB
observe
FediIndex
other
FediIndex
observe
FediList
other
FediList
observe
Fedineko
other
Fedineko
observe
fedistatsCrawler
other
fedistatsCrawler
observe
fedoraplanet
other
fedoraplanet
observe
FedReporter
other
Fed Reporter
FedReporter
observe
FedReporter Bot for FFIEC
seo crawler
Fed Reporter, Inc.
challenge
FedReporterDataBot
ai assistant
Fed Reporter
FedReporterDataBot
observe
Feedbin
social media
Feedbin
Feedbin
allow
feedbot
ai assistant
WordPress
feedbot
observe
FeedBurner
ai assistant
FeedBurner
FeedBurner
observe
FeedDiscovery
other
FeedDiscovery
observe
Feeder
social media
Really Simple AB
Feeder
allow
feeder.co
ai assistant
Really Simple
feeder.co
observe
Feedfetcher-Google
search engine
Feedfetcher-Google
allow
FeedFlow
ai assistant
FeedFlow
FeedFlow
observe
Feedly
social media
Feedly
Feedly
allow
FeedOtter
social media
FeedOtter
allow
Feedsearch
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
Feedspot
ai assistant
Feedspot
Feedspot
observe
FeedValidator
other
FeedValidator
observe
FeedWind Crawler
social media
Mikle
Feedwind
allow
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe
Fever
other
Fever
observe
FindFiles.net
search engine
FindFiles.net
FindFiles.net
allow
FindITAnswersbot
search engine
IT Influentials
FindITAnswersbot
allow
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
fiperbot
observe
FirecrawlAgent
ai crawler
Mendable / Firecrawl
FirecrawlAgent
observe
FirmlyAI Bot
ai assistant
Firmly
observe
FishBot
ai crawler
FishBot
observe
FIT Feeds Fetcher
social media
follow.it
allow
Flaming
other
Flaming
observe