Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
b2w
other
b2w
observe
BackDoorBot
other
BackDoorBot
observe
BacklinkBot
other
BacklinkBot
observe
BacklinkCrawler
other
BacklinkCrawler
observe
BacklinksExtendedBot
other
BacklinksExtendedBot
observe
Bad Bots (mitchellkrogza)
other
1h4x.com
challenge
Baidu-YunGuanCe
other
Baidu-YunGuanCe
observe
Baiduspider-image
other
Baiduspider-image
observe
Baiduspider-news
other
Baiduspider-news
observe
Baiduspider-video
other
Baiduspider-video
observe
BDCbot
other
BDCbot
observe
BehloolBot
other
BehloolBot
observe
betaBot
other
betaBot
observe
bidswitchbot
other
bidswitchbot
observe
BIGLOTRON
other
BIGLOTRON
observe
binlar
other
binlar
observe
Birdcrawlerbot
other
Birdcrawlerbot
observe
BitBot
other
BitBot
observe
Bitbucket
other
Atlassian
Bitbucket-Webhooks
allow
Black Hole
other
Black
observe
Blekkobot
other
Blekkobot
observe
Bling ERP
other
Bling
Bling ERP
observe
blogmuraBot
other
blogmuraBot
observe
BlowFish
other
BlowFish
observe
BLP_bbot
other
BLP_bbot
observe
BOMForge-DeepCrawler
other
BOMForge-DeepCrawler
observe
Bookmark search tool
other
Bookmark
observe
bot-pge.chlooe.com
other
bot-pge.chlooe.com
observe
Bot.AraTurka.com
other
Bot.AraTurka.com
observe
BotALot
other
BotALot
observe
BotRightHere
other
BotRightHere
observe
BoxcarBot
other
BoxcarBot
observe
br-crawler
other
br-crawler
observe
brainobot
other
brainobot
observe
BrandONbot
other
BrandONbot
observe
brokenlinkcheck.com
other
brokenlinkcheck.com
observe
BrokenLinksBot
other
BrokenLinksBot
observe
BTWebClient
other
BTWebClient
observe
BUbiNG
other
BUbiNG
observe
Buddybot
other
Buddybot
observe
BuiltBotTough
other
BuiltBotTough
observe
Bullseye
other
Bullseye
observe
BunnySlippers
other
BunnySlippers
observe
buzzbot
other
buzzbot
observe
CandidateCrawler
other
CandidateCrawler
observe
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
Captify
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
caveman-hunter
other
caveman-hunter
observe
CC Metadata Scaper
other
observe
cc_bot
other
cc_bot
observe
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
Centurybot
other
Centurybot
observe
changedetection
other
changedetection
observe
ChargeBee
other
ChargeBee
ChargeBee
allow