Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe
ExtractorPro
other
ExtractorPro
observe
Eyeotabot
other
Eyeotabot
observe
EZID
other
EZID
observe
EzLynx
ai assistant
Ezoic
EzLynx
observe
EzoicBot-Nicheiq
ai assistant
Ezoic
EzoicBot-Nicheiq
observe
Ezooms
other
Ezooms
observe
FAB-Scraper
other
FAB-Scraper
observe
FacebookBot
ai scraper
Meta
FacebookBot
observe
Facebot
other
Facebot
observe
Factset_spyderbot
ai crawler
Factset
factset_spyderbot
observe
FairAd Client
other
FairAd
observe
FAST Enterprise Crawler
other
FAST
observe
FAST-WebCrawler
other
FAST-WebCrawler
observe
FastCrawler
other
FastCrawler
observe
FastmailUA
ai assistant
Fastmail
FastmailUA
observe
FaviconAPI
ai assistant
Vemetric
FaviconAPI
observe
FediDB
other
FediDB
observe
FediIndex
other
FediIndex
observe
FediList Agent
other
FediList
observe
Fedineko
other
Fedineko
observe
fedistatsCrawler
other
fedistatsCrawler
observe
fedoraplanet
other
fedoraplanet
observe
FedReporterDataBot
ai assistant
Fed Reporter
FedReporterDataBot
observe
feedbot
ai assistant
WordPress
feedbot
observe
FeedBurner
ai assistant
FeedBurner
FeedBurner
observe
FeedDiscovery
other
FeedDiscovery
observe
feeder.co
ai assistant
Really Simple
feeder.co
observe
FeedFlow
ai assistant
FeedFlow
FeedFlow
observe
FeedopsBot
ai assistant
FeedOps
FeedopsBot
observe
Feedsearch Bot
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
Feedspot
ai assistant
Feedspot
Feedspot
observe
Feedspot SSL asset Proxy
other
observe
FeedValidator
other
FeedValidator
observe
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe
Fever
other
Fever
observe
FFZBot
ai assistant
FrankerFaceZ
FFZBot
observe
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
Fiper
fiperbot
observe
FirecrawlAgent
ai crawler
Mendable / Firecrawl
FirecrawlAgent
observe
FireflyBot
other
FireflyBot
observe
FirmCredBot
ai assistant
Firm Cred
FirmCredBot
observe
FirmlyAI Bot
ai assistant
Firmly
firmlyai-bot
observe
FishBot
ai crawler
FishBot
FishBot
observe
Flaming AttackBot
other
Flaming
observe
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
FleebsBot
other
Fleebs
FleebsBot
observe
FlockWP
ai assistant
FlockWP
FlockWP
observe
signed
FlowIQLabsBot
ai assistant
FlowIQ Labs
FlowIQLabsBot
observe
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
ForestEngine
ai assistant
Forest
ForestEngine
observe
foureff
ai assistant
BITS - Benedikt IT Services e.U.
observe
signed