Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
290 results · AI Assistant
CINC
Cincraw
Wikimedia
Citoid
Anthropic
Claude-User
Cledara
CledaraBot
Clickagy
Clickagy
ClickUp
ClickUpLinkUnfurler
Cloudflare
cloudflare-browser-rendering
Cloudflare
Collapsify
composer
Amazon
contxbot
OurFamilyWizard
Cozi-iCalendar-FeedReader
Current
Current
DeckAgent
Devin AI
Devin
dlvr.it
dlvr.it
Publisher Discovery
Dragonfly
Drift Security
drift-agent
Copyright Clearance Center
ds9
DuckDuckGo
DuckAssistBot
DoubleVerify
DVbot
easybill
easybill-ImportManager
codire GmbH
EasyScan
EcoVadis
EcoVadisSustainabilityBot
Techmeme
EpivozCrawler
Headline
ev-crawler
Evernote
EvernoteRichLinkBot
Ezoic
EzLynx
Ezoic
EzoicBot-Nicheiq
Meta
facebookexternalhit
Fastmail
FastmailUA
Fed Reporter
FedReporterDataBot
WordPress
feedbot
FeedBurner
FeedBurner
Really Simple
feeder.co
FeedFlow
FeedFlow
Feedspot
Feedspot
Firmly
firmlyai-bot
FlockWP
FlockWP
BITS - Benedikt IT Services e.U.
FreshRSS
FreshRSS
Friendica
Friendica
GeistHaus
GeistHaus-PageFetcher
Gemini-Deep-Research
Goodreads
Goodreads
Google-adstxt
Google-Agent
Google-PageRenderer
Google-Read-Aloud
GoogleAgent-URLContext
GoogleDocs
GoogleMessages
GoogleProducer
Grammarly
Grammarly
Grapeshot
GrapeshotCrawler
Groovinads
GroovinaAdsbot
CLERK SAS
Haiku-SearchBot