Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
114 results · Search Engine
Luxxlebot
search engine
Luxxle
Luxxlebot
allow
lyonl
search engine
Lyonl
lyonl
allow
lyonl-crawler
search engine
Lyonl
lyonl-crawler
allow
MagiBot
search engine
Peak Labs
MagiBot
allow
Marginalia
search engine
Marginalia
Marginalia
allow
Marginalia Search
search engine
Marginalia
allow
Mars Finder
search engine
Mars Flag
Mars
allow
Mojeek
search engine
Mojeek
MojeekBot
allow
MRGbot
search engine
MRG Web Services srl
MRGbot
allow
MSN
search engine
Microsoft
msnbot
allow
msnbot-media
search engine
Microsoft
msnbot-media
allow
mwmbl
search engine
Mwmbl
mwmbl
allow
Neevabot
search engine
Neeva
Neevabot
allow
NestDaddyBot
search engine
NestDaddy
NestDaddyBot
allow
NewsNow
search engine
NewsNowUK
NewsNow
allow
Notabot
search engine
Helpfeel
Notabot
allow
Potions
search engine
Potions
Potions
allow
Qwantbot
search engine
Qwant
Qwantbot
allow
Qwantbot-official
search engine
Qwant
Qwantbot-official
allow
Qwantify
search engine
Qwant
Qwantify
allow
Scrubby
search engine
Scrub the Web
Scrubby
allow
search.marginalia.nu
search engine
Marginalia
search.marginalia.nu
allow
Seekport
search engine
SISTRIX
Seekport
allow
SeekportBot
search engine
SISTRIX
SeekportBot
allow
SemanticScholarBot
search engine
Ai2
SemanticScholarBot
allow
Seznam
search engine
Seznam
Seznam
allow
SeznamBot
search engine
Seznam
SeznamBot
allow
SiteSearch360
search engine
SEMKNOX GmbH
SiteSearch360
allow
SMTnet PM Bot
search engine
SMTnet
SMTnetPMBot
allow
Sogou
search engine
Sogou
Sogou
allow
SokoCrawler
search engine
eXperience Tech
SokoCrawler
allow
stepstoneCrawlBot
search engine
StepStone
stepstoneCrawlBot
allow
StractBot
search engine
Stract
StractBot
allow
Teoma
search engine
Ask
Teoma
allow
Terracotta
search engine
Ceramic
TerraCotta
allow
TinEye
search engine
TinEye
TinEye
allow
TinEye-bot
search engine
TinEye
TinEye-bot
allow
TinEye-bot-live
search engine
TinEye
TinEye-bot-live
allow
TinEye-Web
search engine
TinEye
TinEye-Web
allow
Toutiao
search engine
Bytedance
allow
WepchSearchEngine
search engine
Wepch
WepchSearchEngine
allow
Y!J-BRZ/YATSHA
search engine
Yahoo
Y!J-BRZ/YATSHA
allow
yacybot
search engine
YaCy
yacybot
allow
Yahoo Slurp
search engine
Yahoo
allow
YandexBot
search engine
Yandex
yandex
allow
YandexImages
search engine
Yandex
YandexImages
allow
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
YandexVertis
search engine
Yandex
YandexVertis
allow
Yeti
search engine
Naver
Yeti
allow
YisouSpider
search engine
Yisou
YisouSpider
allow
YouBot
search engine
You.com
YouBot
allow
ZoomInfo
search engine
ZoomInfo
ZoominfoBot
allow
ZumBot
search engine
Zum Internet Corp
ZumBot
allow