Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
eMoney Advisor
seo crawler
eMoney Advisor
eMoneyBot
challenge
epicbot
other
epicbot
observe
Epivoz Crawler
seo crawler
Techmeme
challenge
EpivozCrawler
ai assistant
Techmeme
EpivozCrawler
observe
eRepublik.tools
monitoring
Sebastian Foth - Software Solutions
eRepublik.tools
allow
eright
other
eright
observe
ErisBot
other
ErisBot
observe
ERNIEBot
ai scraper
Baidu
ERNIEBot
observe
EroCrawler
other
EroCrawler
observe
EtaoSpider
other
EtaoSpider
observe
europarchive.org
other
europarchive.org
observe
ev-crawler
ai assistant
Headline
ev-crawler
observe
evc-batch
other
evc-batch
observe
EvernoteRichLinkBot
ai assistant
Evernote
EvernoteRichLinkBot
observe
everyfeed-spider
other
everyfeed-spider
observe
EveryoneSocialBot
other
EveryoneSocialBot
observe
EvoUptimeBot
monitoring
Evo Agency Ltd.
EvoUptimeBot
allow
ExaBot
ai data provider
Exa
ExaBot
observe
ExaSearchBot
ai crawler
Exa
ExaSearchBot
observe
Exodus
other
Exodus
ExodusMovement
allow
Expanse
security
Palo Alto Networks
Expanse
allow
Experibot
other
Experibot
observe
expo-research-bot
other
expo-research-bot
observe
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe
ExtractorPro
other
ExtractorPro
observe
Eyeotabot
other
Eyeotabot
observe
EZID
other
EZID
observe
EzLynx
ai assistant
Ezoic
EzLynx
observe
EzoicBot
seo crawler
Ezoic Inc
EzoicBot
challenge
EzoicBot-Nicheiq
ai assistant
Ezoic
EzoicBot-Nicheiq
observe
Ezooms
other
Ezooms
observe
FAB-Scraper
other
FAB-Scraper
observe
FacebookBot
ai scraper
Meta
FacebookBot
observe
FacebookExternalHit
social media
Meta
facebook
allow
facebookexternalhit
ai assistant
Meta
facebookexternalhit
observe
Facebot
other
Facebot
observe
Factset_spyderbot
ai crawler
Factset
factset_spyderbot
observe
FairAd Client
other
FairAd
observe
FAST Enterprise Crawler
other
FAST
observe
FAST-WebCrawler
other
FAST-WebCrawler
observe
FastDAST
security
Black Duck
FastDAST
allow
Fastmail Bot
social media
Fastmail
Fastmail Bot
allow
FastmailUA
ai assistant
Fastmail
FastmailUA
observe
FDL Stats Bots
monitoring
FTW Entertainment LLC
FDL
allow
Fedicabot
seo crawler
Fedica
Fedicabot
challenge
FediDB
other
FediDB
observe
FediIndex
other
FediIndex
observe
FediList Agent
other
FediList
observe
Fedineko
other
Fedineko
observe
fedistatsCrawler
other
fedistatsCrawler
observe
fedoraplanet
other
fedoraplanet
observe
FedReporter Bot for FFIEC
seo crawler
Fed Reporter, Inc.
FedReporterDataBot
challenge
FedReporterDataBot
ai assistant
Fed Reporter
FedReporterDataBot
observe
Feedbin
social media
Feedbin
Feedbin
allow
feedbot
ai assistant
WordPress
feedbot
observe
FeedBurner
ai assistant
FeedBurner
FeedBurner
observe
FeedDiscovery
other
FeedDiscovery
observe
Feeder
social media
Really Simple AB
Feeder
allow
feeder.co
ai assistant
Really Simple
feeder.co
observe