Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
BacklinkBot
other
BacklinkBot
observe
BacklinkCrawler
other
BacklinkCrawler
observe
BacklinksExtendedBot
other
BacklinksExtendedBot
observe
Baidu-YunGuanCe
other
Baidu-YunGuanCe
observe
Baiduspider-image
other
Baiduspider-image
observe
Baiduspider-news
other
Baiduspider-news
observe
Baiduspider-video
other
Baiduspider-video
observe
BDCbot
other
BDCbot
observe
bedrockbot
ai scraper
Amazon
bedrockbot
observe
BehloolBot
other
BehloolBot
observe
BestChange
ai assistant
BestChange
BestChange
observe
betaBot
other
betaBot
observe
Bibliotheque Nacional de France Crawler
ai scraper
Bibliothèque nationale de France
bnf.fr_bot
observe
bidswitchbot
other
bidswitchbot
observe
Big Sur AI
ai crawler
Big Sur AI
bigsur.ai
observe
BIGLOTRON
other
BIGLOTRON
observe
bigsur.ai
ai assistant
Big Sur AI
bigsur.ai
observe
binlar
other
binlar
observe
Birdcrawlerbot
other
Birdcrawlerbot
observe
BitBot
other
BitBot
observe
bitlybot
ai assistant
Bitly
bitlybot
observe
BitSightBot
ai assistant
Bitsight
BitSightBot
observe
Black Hole
other
Black
observe
Blackboard
ai assistant
Anthology
Blackboard
observe
Blekkobot
other
Blekkobot
observe
Bling ERP
other
Bling
Bling ERP
observe
BlingERP
ai assistant
Bling
BlingERP
observe
Bloglines
ai assistant
Ask
Bloglines
observe
blogmuraBot
other
blogmuraBot
observe
BlowFish
other
BlowFish
observe
BLP_bbot
other
BLP_bbot
observe
Bluesky
ai assistant
Bluesky
Bluesky
observe
BomboraBot
ai assistant
Bombora
BomboraBot
observe
BOMForge-DeepCrawler
other
BOMForge-DeepCrawler
observe
Bookmark search tool
other
Bookmark
observe
BorderxBot
ai crawler
Borderxlab (Nubestore AI)
BorderxBot
observe
bot-pge.chlooe.com
other
bot-pge.chlooe.com
observe
Bot.AraTurka.com
other
Bot.AraTurka.com
observe
BotALot
other
BotALot
observe
BotRightHere
other
BotRightHere
observe
BoxcarBot
other
BoxcarBot
observe
br-crawler
other
br-crawler
observe
brainobot
other
brainobot
observe
BrandONbot
other
BrandONbot
observe
BrandVerity
ai assistant
BrandVerity
BrandVerity
observe
Brandwatch
ai crawler
Brandwatch
magpie-crawler
observe
Bravebot
ai crawler
Brave Software, Inc.
Bravebot
observe
Brightbot
ai data provider
Bright Data
Brightbot
observe
brokenlinkcheck.com
other
brokenlinkcheck.com
observe
BrokenLinksBot
other
BrokenLinksBot
observe
Browserbase
ai assistant
Browserbase
Browserbase
observe
signed
BTWebClient
other
BTWebClient
observe
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BUbiNG
other
BUbiNG
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buddybot
other
Buddybot
observe
Bugsnag script fetcher
ai coding agent
Bugsnag
observe
BuiltBotTough
other
BuiltBotTough
observe
Bullseye
other
Bullseye
observe