Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
118 results · Search Engine
Qihoo 360
360Spider
Alexandria.org
Alexa
Amazon
Alexandria.org
AlexandriaOrgBot
Algolia
Algolia
AllAfrica Global Media
AllAfrica
RSSMicro
Atom
Baidu
Baidu-ADS
Baidu
Baiduspider
Baidu
Baiduspider-render
Microsoft
bingbot
Microsoft
bingbot
Cloudflare
CloudflareWebScanner
Cludo
cludo.com
Coccoc
coccocbot
Coc Coc
coccocbot-image
Coc Coc
coccocbot-web
Coveo
Coveobot
Freespoke
crawler.freespoke.com
Crawlson
Crawlson
Dataprovider.com
Dataprovider
Dataprovider.com
Dataprovider
Daum
Daum
DocSpot
DocSpotBot
DuckDuckGo
DuckDuckBot
DuckDuckGo
DuckDuckGo-Favicons-Bot
Feedfetcher-Google
FindFiles.net
FindFiles.net
IT Influentials
FindITAnswersbot
Freespoke
crawler.freespoke.com
Freespoke
FreespokeCrawler
Squiz - FunnelBack
Funnelback
Fynd
fyndbot
Fynd
FyndSearchEngine-Crawler
FyndSearchEngine-ReCrawler
Geedo
GeedoProductSearch
Gigablast
Gigabot
Googlebot-Image
Google Scholar
Googlebot-Video
Google-Flights-Search-Spackle
google-xrawler
Googlebot
Googlebot-IA
Googlebot-Image
Googlebot-Mobile
Googlebot-News
Googlebot-Video
Greppr
Greppr
Qihoo 360
HaosouSpider
Hype Machine
Hype
Babbar
IbouBot
Intelligence X
intelx.io_bot
JamBot
Jambot
Jobiglo
Jobiglo-Bot
JobsWithGPT
jobswithgptcom-bot
Jooble
Jooblebot
Kagi
Kagi
Kagibot