Skip to main content
Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

118 results · Search Engine

Level9SearchBot search engine

Level 9

Level9SearchBot
allow
Linespider search engine

LINE Corporation

Linespider
allow
Luxxlebot search engine

Luxxle

Luxxlebot
allow
lyonl search engine

Lyonl

lyonl
allow
lyonl-crawler search engine

Lyonl

lyonl-crawler
allow
MagiBot search engine

Peak Labs

MagiBot
allow
Marginalia Search search engine

Marginalia

search.marginalia.nu
allow
Mars Finder search engine

Mars Flag

Mars
allow
Mojeek search engine

Mojeek

MojeekBot
allow
MRGbot search engine

MRG Web Services srl

MRGbot
allow
MSN search engine

Microsoft

msnbot
allow
msnbot-media search engine

Microsoft

msnbot-media
allow
mwmbl search engine

Mwmbl

mwmbl
allow
Neevabot search engine

Neeva

Neevabot
allow
NestDaddyBot search engine

NestDaddy

NestDaddyBot
allow
NewsNow search engine

NewsNowUK

NewsNow
allow
Notabot search engine

Helpfeel

Notabot
allow
Potions search engine

Potions

Potions
allow
Qwantbot search engine

Qwant

Qwantbot
allow
Qwantbot-official search engine

Qwant

Qwantbot-official
allow
Qwantify search engine

Qwant

Qwantify
allow
Scrubby search engine

Scrub the Web

Scrubby
allow
search.marginalia.nu search engine

Marginalia

search.marginalia.nu
allow
Seekport search engine

SISTRIX

Seekport
allow
SeekportBot search engine

SISTRIX

SeekportBot
allow
SemanticScholarBot search engine

Ai2

SemanticScholarBot
allow
Seznam search engine

Seznam

Android(10);cz.seznam.sbrowser;12.6.0
allow
SeznamBot search engine

Seznam

SeznamBot
allow
SiteSearch360 search engine

SEMKNOX GmbH

SiteSearch360
allow
SMTnet PM Bot search engine

SMTnet

SMTnetPMBot
allow
Sogou Push Spider search engine

Sogou

Sogou Push Spider
allow
Sogou web spider search engine

Sogou

allow
SokoCrawler search engine

eXperience Tech

SokoCrawler
allow
stepstoneCrawlBot search engine

StepStone

stepstoneCrawlBot
allow
StractBot search engine

Stract

StractBot
allow
Teoma search engine

Ask

Teoma
allow
Terracotta search engine

Ceramic

TerraCotta
allow
TinEye search engine

TinEye

TinEye
allow
TinEye-bot search engine

TinEye

TinEye-bot
allow
TinEye-bot-live search engine

TinEye

TinEye-bot-live
allow
TinEye-Web search engine

TinEye

TinEye-Web
allow
Toutiao search engine

Bytedance

zhanzhang.toutiao.com
allow
WepchSearchEngine search engine

Wepch

WepchSearchEngine
allow
Y!J-BRZ/YATSHA crawler search engine

Yahoo

allow
yacybot search engine

YaCy

yacybot
allow
Yahoo Slurp search engine

Yahoo

allow
Yahoo! Slurp search engine

Yahoo

allow
YandexBot search engine

Yandex

http://yandex.com/bots
allow rDNS
YandexImages search engine

Yandex

YandexImages
allow rDNS
YandexMobileBot search engine

Yandex

YandexMobileBot
allow rDNS
YandexRenderResourcesBot search engine

Yandex

YandexRenderResourcesBot
allow rDNS
YandexVertis search engine

Yandex

YandexVertis
allow rDNS
Yeti search engine

Naver

Yeti
allow
Yeti by Naver search engine

Naver

allow
YisouSpider search engine

Yisou

YisouSpider
allow
YouBot search engine

You.com

YouBot
allow
ZoomInfo search engine

ZoomInfo

ZoominfoBot
allow
ZumBot search engine

Zum Internet Corp

ZumBot
allow
← Prev 2 / 2 Next →