Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
everyfeed-spider
other
everyfeed-spider
observe
EveryoneSocialBot
other
EveryoneSocialBot
observe
Exodus
other
Exodus
ExodusMovement
allow
Experibot
other
Experibot
observe
expo-research-bot
other
expo-research-bot
observe
ExposureWatch-Benchmark
other
ExposureWatch-Benchmark
observe
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe
ExtractorPro
other
ExtractorPro
observe
Eyeotabot
other
Eyeotabot
observe
EZID
other
EZID
observe
Ezooms
other
Ezooms
observe
FAB-Scraper
other
FAB-Scraper
observe
Facebot
other
Facebot
observe
FairAd Client
other
FairAd
observe
FAST Enterprise Crawler
other
FAST
observe
FAST-WebCrawler
other
FAST-WebCrawler
observe
FastCrawler
other
FastCrawler
observe
FediDB
other
FediDB
observe
FediIndex
other
FediIndex
observe
FediList Agent
other
FediList
observe
Fedineko
other
Fedineko
observe
fedistatsCrawler
other
fedistatsCrawler
observe
fedoraplanet
other
fedoraplanet
observe
FeedDiscovery
other
FeedDiscovery
observe
Feedsearch Bot
other
Feedsearch
observe
Feedsearch-Crawler
other
Feedsearch-Crawler
observe
Feedspot SSL asset Proxy
other
observe
FeedValidator
other
FeedValidator
observe
FemtosearchBot
other
FemtosearchBot
observe
fetch-nodeinfo-bot
other
fetch-nodeinfo-bot
observe
Fever
other
Fever
observe
findlink
other
findlink
observe
findthatfile
other
findthatfile
observe
findxbot
other
findxbot
observe
fiperbot
other
Fiper
fiperbot
observe
FireflyBot
other
FireflyBot
observe
Flaming AttackBot
other
Flaming
observe
Flamingo_SearchEngine
other
Flamingo_SearchEngine
observe
Flashy-PHP/1.0.0
other
Flashy
Flashy-PHP
allow
FleebsBot
other
Fleebs
FleebsBot
observe
fluffy
other
fluffy
observe
fluid
other
fluid
observe
Flyriverbot
other
Flyriverbot
observe
Foobot
other
Foobot
observe
fr-crawler
other
fr-crawler
observe
FreeWebMonitoring SiteChecker
other
FreeWebMonitoring
observe
Freshbot
other
Freshbot
observe
Freshping
other
Freshworks
Freshping
allow
fuelbot
other
fuelbot
observe
FusionBot
other
FusionBot
observe
Fyrebot
other
Fyrebot
observe
g00g1e.net
other
g00g1e.net
observe
G2 Web Services
other
observe
g2reader-bot
other
g2reader-bot
observe
Gaisbot
other
Gaisbot
observe
GarlikCrawler
other
GarlikCrawler
observe
GeedoShopProductFinder
other
GeedoShopProductFinder
observe
rDNS
GeneralCrawlBot
other
GeneralCrawlBot
observe
Genieo
other
Genieo
observe