Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
268 results · AI Assistant
Cloudflare
Collapsify
Amazon
contxbot
OurFamilyWizard
Cozi-iCalendar-FeedReader
Current
Current
Devin AI
Devin
dlvr.it
dlvr.it
Publisher Discovery
Dragonfly
Drift Security
Copyright Clearance Center
ds9
DuckDuckGo
DuckAssistBot
DoubleVerify
DVbot
codire GmbH
EcoVadis
EcoVadisSustainabilityBot
Techmeme
EpivozCrawler
Headline
ev-crawler
Evernote
EvernoteRichLinkBot
Ezoic
EzLynx
Ezoic
EzoicBot-Nicheiq
Fastmail
FastmailUA
Fed Reporter
FedReporterDataBot
WordPress
feedbot
FeedBurner
FeedBurner
Really Simple
feeder.co
FeedFlow
FeedFlow
Feedspot
Feedspot
Firmly
FlockWP
BITS - Benedikt IT Services e.U.
FreshRSS
FreshRSS
Friendica
Friendica
GeistHaus
GeistHaus-PageFetcher
Gemini-Deep-Research
Goodreads
Goodreads
Google-adstxt
Google-Agent
GoogleAgent-URLContext
GoogleDocs
GoogleProducer
Grammarly
Grammarly
Grapeshot
GrapeshotCrawler
Groovinads
GroovinaAdsbot
CLERK SAS
Hatena
Hatena::RUssia::Crawler
Hatena
HatenaBlog-bot
Check Point Software Technologies Ltd.
Henry Labs
HubSpot
HubSpot-FeedFetcher
Integral Ad Science
ias-ie
SpazioDati
IDG/IT
IFTTT
IFTTT
Hive
ImagesiftBot
Inoreader
Inoreader
Innologica
inoreader.com
Instapaper
Instaparser
OneTag
InteractiveAdvertisingBureau
IONOS
IonCrawl
JusProg
Jugendschutzprogramm-Crawler
Kagi
kagi-fetcher