Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
CINC
Cincraw
Wikimedia
Citoid
Anthropic
Claude-User
Cledara
CledaraBot
Clickagy
Clickagy
ClickUp
ClickUpLinkUnfurler
Cloudflare
cloudflare-browser-rendering
Cloudflare
Collapsify
CompanyMapBot
composer
ContentStudio
ContentStudio
ContentStudio
ContentStudio-Feed-Fetcher
ContextWebCom-AdsTxt
Amazon
contxbot
OurFamilyWizard
Cozi-iCalendar-FeedReader
Current
Current
CypressGrowthDiligence
DeckAgent
Devin AI
Devin
dlvr.it
dlvr.it
DRM3
DomainDrift
Publisher Discovery
Dragonfly
Drift Security
drift-agent
Copyright Clearance Center
ds9
DuckDuckGo
DuckAssistBot
DoubleVerify
DVbot
Dwell
dwelltime-bot
codire GmbH
EasyScan
EcoVadis
EcoVadisSustainabilityBot
ElevenLabs
ElevenlabsBot
Techmeme
EpivozCrawler
Headline
ev-crawler
Evernote
EvernoteRichLinkBot
Exa
ExaUser
Ezoic
EzLynx
Ezoic
EzoicBot-Nicheiq
Fastmail
FastmailUA
Vemetric
FaviconAPI
Fed Reporter
FedReporterDataBot
WordPress
feedbot
FeedBurner
FeedBurner
Really Simple
feeder.co
FeedFlow
FeedFlow
FeedOps
FeedopsBot
Feedspot
Feedspot
FrankerFaceZ
FFZBot
Firm Cred
FirmCredBot
Firmly
firmlyai-bot
FlockWP
FlockWP
FlowIQ Labs
FlowIQLabsBot
Forest
ForestEngine
BITS - Benedikt IT Services e.U.
FreshRSS
FreshRSS
Friendica
Friendica
GeistHaus
GeistHaus-PageFetcher
Gemini-Deep-Research
getdomaindata.com
getdomaindata
Ghost
GlobRadar
GlobRadarBot