Skip to main content
Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear
All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

134 results · Search Engine

IbouBot search engine

Babbar

IbouBot
allow
intelx.io_bot search engine

Intelligence X

intelx.io_bot
allow
Jambot search engine

JamBot

Jambot
allow
Jobiglo-Bot search engine

Jobiglo

Jobiglo-Bot
allow
jobswithgptcom-bot search engine

JobsWithGPT

jobswithgptcom-bot
allow
Jooblebot search engine

Jooble

Jooblebot
allow
Kagi Bot search engine

Kagi

allow
Kagibot search engine

Kagi

Kagibot
allow rDNS
Level9SearchBot search engine

Level 9

Level9SearchBot
allow
Linespider search engine

LINE Corporation

Linespider
allow
LohiSoftBot search engine

LohiSoft

LohiSoftBot
allow
Luxxlebot search engine

Luxxle

Luxxlebot
allow
lyonl search engine

Lyonl

lyonl
allow
lyonl-crawler search engine

Lyonl

lyonl-crawler
allow
MagiBot search engine

Peak Labs

MagiBot
allow
Marginalia Search search engine

Marginalia

search.marginalia.nu
allow
Mars Finder search engine

Mars Flag

Mars
allow
MetaCrawler search engine
MetaCrawler
allow
Mojeek search engine

Mojeek

MojeekBot
allow
MRGbot search engine

MRG Web Services srl

MRGbot
allow
MSN search engine

Microsoft

msnbot
allow
msnbot-media search engine

Microsoft

msnbot-media
allow
mwmbl search engine

Mwmbl

mwmbl
allow
NagetBot search engine

Naget

NagetBot
allow
Neevabot search engine

Neeva

Neevabot
allow
NestDaddyBot search engine

NestDaddy

NestDaddyBot
allow
NewsNow search engine

NewsNowUK

NewsNow
allow
Notabot search engine

Helpfeel

Notabot
allow
PageSourceSearchBot search engine

PageSourceSearch

PageSourceSearchBot
allow
Potions search engine

Potions

Potions
allow
PublicWWWBot search engine

PublicWWW

PublicWWWBot
allow
PurityBot search engine

Purili

PurityBot
allow
Qwantbot search engine

Qwant

Qwantbot
allow
Qwantbot-official search engine

Qwant

Qwantbot-official
allow
Qwantify search engine

Qwant

Qwantify
allow
Scrubby search engine

Scrub the Web

Scrubby
allow
search.marginalia.nu search engine

Marginalia

search.marginalia.nu
allow
Seekport search engine

SISTRIX

Seekport
allow
SeekportBot search engine

SISTRIX

SeekportBot
allow
SemanticScholarBot search engine

Ai2

SemanticScholarBot
allow
Seznam search engine

Seznam

Android(10);cz.seznam.sbrowser;12.6.0
allow
SeznamBot search engine

Seznam

SeznamBot
allow
SiteSearch360 search engine

SEMKNOX GmbH

SiteSearch360
allow
slopsearch-macos-crawler search engine
slopsearch-macos-crawler
allow
SMTnet PM Bot search engine

SMTnet

SMTnetPMBot
allow
Sogou Push Spider search engine

Sogou

Sogou Push Spider
allow
Sogou web spider search engine

Sogou

allow
SokoCrawler search engine

eXperience Tech

SokoCrawler
allow
stepstoneCrawlBot search engine

StepStone

stepstoneCrawlBot
allow
StractBot search engine

Stract

StractBot
allow
Teoma search engine

Ask

Teoma
allow
Terracotta search engine

Ceramic

TerraCotta
allow
TinEye search engine

TinEye

TinEye
allow
TinEye-bot search engine

TinEye

TinEye-bot
allow
TinEye-bot-live search engine

TinEye

TinEye-bot-live
allow
TinEye-Web search engine

TinEye

TinEye-Web
allow
Toutiao search engine

Bytedance

zhanzhang.toutiao.com
allow
VivianCrawler search engine

Evil Source Platform

VivianCrawler
allow
WepchSearchEngine search engine

Wepch

WepchSearchEngine
allow
What10Bot search engine
What10Bot
allow
← Prev 2 / 3 Next →