Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
nyt_scraping
seo crawler
The New York Times
nyt_scraping
challenge
OAI-AdsBot
seo crawler
OpenAI
OAI-AdsBot
challenge
OAI-SearchBot
ai crawler
OpenAI
OAI-SearchBot
observe
Observer
security
Observer
Observer
allow
Odin
security
ODIN
Odin
allow
OdklBot
ai assistant
VK
OdklBot
observe
officestorebot
other
officestorebot
observe
Offline
seo crawler
MetaProducts
Offline
challenge
Oh
ai coding agent
Oh Dear
Oh
observe
Oh Dear
monitoring
Oh Dear
OhDear
allow
OhDear.app
ai coding agent
Oh Dear
OhDear.app
observe
oii-research
other
oii-research
observe
OKX-dolphin-crawler
security
OKX
allow
omgili
ai scraper
Webz.io
omgili
observe
Omgilibot
other
Omgilibot
observe
Omnisend
other
Omnisend
allow
OnCrawl
seo crawler
Oncrawl
OnCrawl
challenge
OneTrust
security
OneTrust
OneTrust
allow
Onetrust CMP Scanner
monitoring
OneTrust, LLC.
Onetrust
allow
Online Webceo Bot
seo crawler
Online Webceo
online-webceo-bot
challenge
OnlineOrNot Monitor
monitoring
OnlineOrNot
OnlineOrNot
allow
OnlineOrNot.com_bot
ai coding agent
OnlineOrNot
OnlineOrNot.com_bot
observe
Openbot
other
Openbot
observe
opencode
ai coding agent
Anomaly
opencode
observe
openfind
other
openfind
observe
Opengraph Bot
social media
Opengraph
allow
OpenGraph.io
ai assistant
Opengraph
OpenGraph.io
observe
OpenGraphCheck
other
OpenGraphCheck
observe
OpenHoseBot
other
OpenHoseBot
observe
OpenindexSpider
seo crawler
Openindex
OpenindexSpider
challenge
OpenRSS
social media
OpenRSS
OpenRSS
allow
OpenTheBoxBot
other
OpenTheBoxBot
observe
opinion-tracker
other
opinion-tracker
observe
Oracle
other
Oracle
observe
OrangeBot
other
OrangeBot
observe
Orbbot
other
Orbbot
observe
Orlo Link Preview
social media
Orlo
Orlo-LinkPreview
allow
Orthogaffe
other
Orthogaffe
observe
OurFamilyWizard
social media
OurFamilyWizard
allow
outboundbuddy-enrichment
ai assistant
Outbound Buddy
outboundbuddy-enrichment
observe
Outbrain
seo crawler
Outbrain
outbrain
challenge
OutclicksBot
other
OutclicksBot
observe
OutsellURLValidator
monitoring
Outsell Corporation
OutsellURLValidator
allow
Overcast
social media
Overcast Radio
Overcast
allow
Owler
ai assistant
Owler
Owler
observe
OWLer-W
other
OWLer-W
observe
Ozon Web Grabber
social media
Ozon
Ozon
allow
Paddle Webhooks
other
Paddle
allow
PaesslerCloudBot
monitoring
Paessler
allow
page-preview-tool
ai assistant
SalesViewer
page-preview-tool
observe
Page2RSS
other
Page2RSS
observe
PagePeeker
ai assistant
PagePeeker
PagePeeker
observe
PageThing
other
PageThing
observe
Pandalytics
ai assistant
Domainsbot
Pandalytics
observe
PanguBot
ai scraper
Huawei
PanguBot
observe
Panopta
monitoring
Panopta
Panopta
allow
panscient.com
ai assistant
Panscient
panscient.com
observe
Paqlebot
other
Paqle
Paqlebot
observe
parisbot
other
parisbot
observe
Parse.ly
monitoring
Parse.ly
parse.ly
allow