Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Amazon
bedrockbot
BehloolBot
BestChange
BestChange
BestChange
BestChange Bot
betaBot
Better Stack
Better
Bibliothèque nationale de France
bnf.fr_bot
bidswitchbot
Big Sur AI
bigsur.ai
BIGLOTRON
BigScoots
BigScootsMonitor
Big Sur AI
bigsur.ai
BigUpData
BigUpDataBot
Binary Canary
binarycanary
Microsoft
adidxbot
Microsoft
BingPreview
Microsoft
bingbot
Microsoft
bingbot
binlar
Birdcrawlerbot
BitBot
Atlassian
Bitbucket-Webhooks
Bitly
bitlybot
Bitsight
BitSightBot
British Library
bl.uk_lddc_bot
Black Duck
Black
Black
Anthology
Blackboard
BlackVeil
BlackVeil-Security-Scanner
Blekkobot
WebMeUp
BLEXBot
Bling
Bling ERP
Bling
BlingERP
Blockaid
Blockaid
Ask
Bloglines
blogmuraBot
Blogtrottr
Blogtrottr
BlogVault
BlogVault
BlowFish
BLP_bbot
Bluesky
Bluesky
Bluesky
bne.es_bot
Bibliothèque nationale de France
bnf.fr_bot
KP Software Consult ApS
BoardGamePrices
Bombora
BomboraBot
BOMForge-DeepCrawler
Bookmark
Borderxlab (Nubestore AI)
BorderxBot
bot-pge.chlooe.com
Bot.AraTurka.com
BotALot
Botify
botify
BotRightHere
BoxcarBot
br-crawler
brainobot
BrandONbot
BrandVerity
BrandVerity
Brandwatch
magpie-crawler