Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
electricmonk
other
electricmonk
observe
Elisabot
other
Elisabot
observe
ellisphere
other
ellisphere
observe
EmailCollector
other
EmailCollector
observe
EmailSiphon
other
EmailSiphon
observe
EmailWolf
other
EmailWolf
observe
emetriqContextualBot
other
emetriqContextualBot
observe
epicbot
other
epicbot
observe
EpivozCrawler
ai assistant
Techmeme
EpivozCrawler
observe
eright
other
eright
observe
ErisBot
other
ErisBot
observe
ERNIEBot
ai scraper
Baidu
ERNIEBot
observe
EroCrawler
other
EroCrawler
observe
EtaoSpider
other
EtaoSpider
observe
europarchive.org
other
europarchive.org
observe
ev-crawler
ai assistant
Headline
ev-crawler
observe
evc-batch
other
evc-batch
observe
EvernoteRichLinkBot
ai assistant
Evernote
EvernoteRichLinkBot
observe
everyfeed-spider
other
everyfeed-spider
observe
EveryoneSocialBot
other
EveryoneSocialBot
observe
ExaBot
ai data provider
Exa
ExaBot
observe
ExaSearchBot
ai crawler
Exa
ExaSearchBot
observe
Experibot
other
Experibot
observe
expo-research-bot
other
expo-research-bot
observe
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe
ExtractorPro
other
ExtractorPro
observe
Eyeotabot
other
Eyeotabot
observe
EZID
other
EZID
observe
EzLynx
ai assistant
Ezoic
EzLynx
observe
EzoicBot-Nicheiq
ai assistant
Ezoic
EzoicBot-Nicheiq
observe
Ezooms
other
Ezooms
observe
FAB-Scraper
other
FAB-Scraper
observe
FacebookBot
ai scraper
Meta
FacebookBot
observe
facebookexternalhit
ai assistant
Meta
facebookexternalhit
observe
Facebot
other
Facebot
observe
Factset_spyderbot
ai crawler
Factset
factset_spyderbot
observe
FairAd Client
other
FairAd
observe
FAST Enterprise Crawler
other
FAST
observe
FAST-WebCrawler
other
FAST-WebCrawler
observe
FastmailUA
ai assistant
Fastmail
FastmailUA
observe
FediDB
other
FediDB
observe
FediIndex
other
FediIndex
observe
FediList Agent
other
FediList
observe
Fedineko
other
Fedineko
observe
fedistatsCrawler
other
fedistatsCrawler
observe
fedoraplanet
other
fedoraplanet
observe
FedReporterDataBot
ai assistant
Fed Reporter
FedReporterDataBot
observe
feedbot
ai assistant
WordPress
feedbot
observe
FeedBurner
ai assistant
FeedBurner
FeedBurner
observe
FeedDiscovery
other
FeedDiscovery
observe
feeder.co
ai assistant
Really Simple
feeder.co
observe
FeedFlow
ai assistant
FeedFlow
FeedFlow
observe
Feedsearch Bot
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
Feedspot
ai assistant
Feedspot
Feedspot
observe
Feedspot SSL asset Proxy
other
observe
FeedValidator
other
FeedValidator
observe
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe