Skip to main content
Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked
Clear
All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

187 results · SEO Crawler

Google Digital Asset Links seo crawler

Google

Google
challenge
Google Firebase seo crawler

Google

challenge
Google StoreBot seo crawler

Google

Storebot-Google
challenge
Google-AdWords-Express seo crawler

Google

Google-AdWords-Express
challenge
Google-Adwords-Instant seo crawler

Google

Google-Adwords-Instant
challenge
Google-Display-Ads-Bot seo crawler

Google

Google-Display-Ads-Bot
challenge
Google-Structured-Data-Testing seo crawler

Google

Google-Structured-Data-Testing
challenge
Google-TravelAds-Live seo crawler

Google

challenge
Grapeshot seo crawler

Grapeshot

grapeshot
challenge
Groovinads seo crawler

Groovinads

challenge
HelloworkJobPostingBot seo crawler

Hellowork

HelloworkJobPostingBot
challenge
HTTrack seo crawler
HTTrack
challenge
Huckabuy Bot seo crawler

Huckabuy

Huckabot
challenge
hypestat seo crawler

HypeStat

hypestat
challenge
IAS crawler seo crawler

Integral Ad Science

IAS
challenge
Idealo seo crawler

Idealo internet GmbH

challenge
IndeedJobBot seo crawler

Indeed

IndeedJobBot
challenge
James seo crawler

cognitiveSEO

James
challenge
klaviyo seo crawler

Klaviyo

Klaviyo
challenge
KrawlerBot seo crawler

Krawler

KrawlerBot
challenge
LeikiBot seo crawler

DoubleVerify

Leikibot
challenge
LibreCrawl seo crawler
LibreCrawl
challenge
LINE OGP Scraper seo crawler

LINE

challenge
LinkCheck seo crawler

Siteimprove

LinkCheck
challenge
LinkChecker Bot seo crawler

LinkChecker

LinkChecker
challenge
linkchecker.pro seo crawler

LinkChecker

linkchecker.pro
challenge
LinkCheckerBot seo crawler

LinkChecker.pro

LinkCheckerBot
challenge
linkdexbot seo crawler

Linkdex

linkdexbot
challenge
MagnetmeBot seo crawler

Magnet.me

MagnetmeBot
challenge
marketgoo seo crawler

marketgoo

MarketGoo
challenge
MBCrawler seo crawler

SEOptimer

MBCrawler
challenge
MediaMonitoringBot seo crawler

MediaMonitoringBot

MediaMonitoringBot
challenge
Mediatoolkitbot seo crawler

Determ

Determ
challenge
Mediavine Medatada Parser seo crawler

Mediavine

challenge
MegaIndex.ru seo crawler

MegaIndex

MegaIndex.ru
challenge
Meta-ExternalAds seo crawler

Meta

meta-externalads
challenge
MetaInspector seo crawler
MetaInspector
challenge
MetaJobBot seo crawler

METAJob

MetaJobBot
challenge
MgidBot seo crawler

MGID

MgidBot
challenge
MirrorWebCrawler seo crawler

MirrorWeb Ltd

MirrorWebCrawler
challenge
Missinglettr Bot seo crawler

Missinglettr

MissinglettrBot
challenge
MJ12Bot seo crawler

Majestic

MJ12bot
challenge
ModernizeBot seo crawler

ModernizeYourWebsite

ModernizeBot
challenge
MotoMinerBot seo crawler

Motominer

MotoMinerBot
challenge
Moz seo crawler

Moz

Moz
challenge
Moz dotbot seo crawler

Moz

DotBot
challenge
Moz rogerbot seo crawler

Moz

rogerbot
challenge
newspaper seo crawler
newspaper
challenge
NitroBot seo crawler

NitroPack Ltd

Nitro-
challenge
Noorobot seo crawler

Noor Digital Agency AB

noorobot
challenge
NostoCrawlerBot seo crawler

Nosto

NostoCrawlerBot
challenge
Nostra seo crawler

Nostra

challenge
Nutch seo crawler
Nutch
challenge
nyt_scraping seo crawler

The New York Times

nyt_scraping
challenge
OAI-AdsBot seo crawler

OpenAI

OAI-AdsBot
challenge
Offline seo crawler

MetaProducts

Offline
challenge
OnCrawl seo crawler

Oncrawl

OnCrawl
challenge
Online Webceo Bot seo crawler

Online Webceo

online-webceo-bot
challenge
OpenindexSpider seo crawler

Openindex

OpenindexSpider
challenge
Outbrain seo crawler

Outbrain

outbrain
challenge
← Prev 2 / 4 Next →