How to block any crawler
Rules you can paste, for 1,186 agents: nginx, Apache, Cloudflare and WordPress, plus the robots.txt directive where the agent publishes one.
AI Agent
AI Assistant
- A360-Search · Area360
- AASA-Bot · Apple
- ActiveComply · ActiveComply
- adbeat_bot · Adbeat
- AddThis.com · Oracle
- AdkernelTopicCrawler · AdKernel
- admantx · Integral Ad Science
- AdmixerBot · Admixer
- AdsBot-Google · Google
- AdsBot-Google-Mobile · Google
- AffsignalCrawler · Affsignal
- AGI Agent · AGI, Inc
- AI2Bot-DeepResearchEval · Ai2
- aiHitBot · aiHit
- AllAfrica · AllAfrica Global Media
- AllAfrica NewsBot · AllAfrica
- Amazing-SearchBot · Amazing
- AmazingTest-SearchBot · Amazing
- Amazon Bedrock AgentCore Browser (AP Northeast) · Amazon
- Amazon Bedrock AgentCore Browser (AP South) · Amazon
- Amazon Bedrock AgentCore Browser (AP Southeast 1) · Amazon
- Amazon Bedrock AgentCore Browser (AP Southeast) · Amazon
- Amazon Bedrock AgentCore Browser (EU Central 1) · Amazon
- Amazon Bedrock AgentCore Browser (EU West 1) · Amazon
- Amazon Bedrock AgentCore Browser (US East 1) · Amazon
- Amazon Bedrock AgentCore Browser (US East 2) · Amazon
- Amazon Bedrock AgentCore Browser (US West 2) · Amazon
- Amazon Buy For Me · Amazon
- Amazon-Advertising-ad-standards-bot · Amazon
- amazon-QBusiness · Amazon
- AmazonProductDiscovery · Amazon
- Amzn-User · Amazon
- AndersPinkBot · Anders Pink
- Apify Website Content Crawler · Apify
- ArchiveBot · Wikimedia
- ArenaUnfurlBot · Arena
- Assaybot · Index Exchange
- AudigentAdBot · Audigent
- Automattic Analytics Crawler · WordPress
- AwarioBot · Awario
- BestChange · BestChange
- bigsur.ai · Big Sur AI
- bitlybot · Bitly
- BitSightBot · Bitsight
- Blackboard · Anthology
- BlingERP · Bling
- Bloglines · Ask
- Bluesky · Bluesky
- BomboraBot · Bombora
- BrandVerity · BrandVerity
- Browserbase · Browserbase
- BubblesBot · Bubbles
- BublupBot · Bublup
- Buck · Hypefactors
- Buttondown · Buttondown
- CartAI Agentic Commerce as a Service · CartAI
- channable · Channable
- ChatGPT agent · OpenAI
- ChatGPT-User · OpenAI
- CheckMarkNetwork · CheckMark Network
- Cincraw · CINC
- Citoid · Wikimedia
- Claude-User · Anthropic
- Cledara SaaS Management Agent · Cledara
- Clickagy Intelligence Bot · Clickagy
- Clickagy Intelligence Bot v2 · Clickagy
- ClickUpLinkUnfurler · ClickUp
- Cloudflare Browser Run · Cloudflare
- Collapsify · Cloudflare
- Composer
- contxbot · Amazon
- Cozi-iCalendar-FeedReader · OurFamilyWizard
- Current · Current
- Deck Agent
- Devin · Devin AI
- dlvr.it · dlvr.it
- Dragonfly · Publisher Discovery
- Drift Agent · Drift Security
- ds9 · Copyright Clearance Center
- DuckAssistbot · DuckDuckGo
- DVbot · DoubleVerify
- easybill-ImportManager · easybill
- EasyScan · codire GmbH
- EcoVadisSustainabilityBot · EcoVadis
- EpivozCrawler · Techmeme
- ev-crawler · Headline
- EvernoteRichLinkBot · Evernote
- EzLynx · Ezoic
- EzoicBot-Nicheiq · Ezoic
- facebookexternalhit · Meta
- FastmailUA · Fastmail
- FedReporterDataBot · Fed Reporter
- feedbot · WordPress
- FeedBurner · FeedBurner
- feeder.co · Really Simple
- FeedFlow · FeedFlow
- Feedspot · Feedspot
- FirmlyAI Bot · Firmly
- FlockWP · FlockWP
- foureff · BITS - Benedikt IT Services e.U.
- FreshRSS · FreshRSS
- Friendica · Friendica
- GeistHaus-PageFetcher · GeistHaus
- Gemini-Deep-Research · Google
- Goodreads · Goodreads
- Google AdSense · Google
- Google AppsViewer · Google
- Google Web Preview · Google
- Google-adstxt · Google
- Google-Agent · Google
- Google-PageRenderer · Google
- Google-Read-Aloud · Google
- GoogleAgent-URLContext · Google
- GoogleDocs · Google
- GoogleMessages · Google
- GoogleProducer · Google
- Grammarly · Grammarly
- GrapeshotCrawler · Grapeshot
- GroovinaAdsbot · Groovinads
- Haiku-SearchBot · CLERK SAS
- Hatena::RUssia::Crawler · Hatena
- HatenaBlog-bot · Hatena
- hecBot · Check Point Software Technologies Ltd.
- Henry Shopping Agent · Henry Labs
- HubSpot · HubSpot
- HubSpot-FeedFetcher · HubSpot
- ias-ie · Integral Ad Science
- IDG/IT · SpazioDati
- IFTTT · IFTTT
- ImagesiftBot · Hive
- iMessage · Apple
- Innguma · Innguma
- Inoreader · Inoreader
- inoreader.com · Innologica
- Instaparser · Instapaper
- InteractiveAdvertisingBureau · OneTag
- IonCrawl · IONOS
- Jugendschutzprogramm-Crawler · JusProg
- kagi-fetcher · Kagi
- KargoBot · Kargo
- KargoBot-Artemis-Mobile · Kargo
- Kernel · Kernel
- Kimi-User · Moonshot AI
- Known Agent · Known Agents
- KStandBot · URL Classification
- LighthouseBot · Lighthouse
- LightspeedSystemsCrawler · Lightspeed Systems
- LinerBot · Liner
- Link CLI · Link
- linkfluence · Meltwater
- LinkPreview · Interactive32
- LinkWalker · Fortra
- LMArenaUnfurlBot
- Magnet.me-web · Magnet
- magpie-crawler · Brandwatch
- MagpieRSS · SourceForge
- MailChimp · Mailchimp
- Mantisbot · Mantis
- Manus Bot · Manus
- Mastodon · Mastodon
- McontextualBot · MContextual
- Mediapartners-Google · Google
- MediavineMetadataParser · Mediavine
- Mediumbot-MetaTagFetcher · Medium
- Meta-ExternalFetcher · Meta
- Miniflux · Miniflux
- MistralAI-User · Mistral AI
- MixrankBot · MixRank
- MuckRack · Muck Rack
- MuckRackFeedParser · Muck Rack
- NanoInteractive · Nano Interactive
- Nava Labs ASP (Dev) · Nava Labs
- Nekuda Payment Executor · Nekuda
- NetcraftSurveyAgent · Netcraft
- netEstate NE Crawler · netEstate
- Neticle Crawler · Neticle
- NetSeer crawler · NetSeer
- Netvibes · Netvibes
- NeuwoBot · Neuwo
- NewsBlur Feed Fetcher · NewsBlur
- NewsBlur Page Fetcher · NewsBlur
- Newslitbot · Newslit
- NewsRoom.BI · Marfeel
- NextCloud · Nextcloud
- OdklBot · VK
- OpenGraph.io · Opengraph
- Orlo-LinkPreview · Orlo
- outboundbuddy-enrichment · Outbound Buddy
- Owler · Owler
- page-preview-tool · SalesViewer
- PagePeeker · PagePeeker
- Pandalytics · Domainsbot
- panscient.com · Panscient
- PartnerOptimizer-bot · PartnerOptimizer
- Payhawk Invoice Fetching Agent · Payhawk
- Perplexity-User · Perplexity
- Perplexity‑User · Perplexity
- PhindBot · Phind
- Piqo Podsync · Piqo
- PocketParser
- PodchaserParser · Podchaser
- Podimo · Podimo
- Poggio-Citations · Poggio
- PressEngineBot · PressEngine
- productsup.io/crawler · Productsup
- QAtechBot · QA.tech
- qcbot · QUIC.cloud
- Quora-Bot · Quora
- QuuuBot · Quuu
- Qwarrybot · Qwarry
- RankurBot · Rankur
- Readwise · Readwise
- Readybot.io · Readybot
- Redactus Proxy Service · Redactus
- redditbot · Reddit
- RSS.Social · RSS.Social
- RyeBot · Rye
- ScavoBot · Scavo
- scoop.it · Meltwater
- scoopit-crawler · Scoop.it
- Scope3 · Scope3
- scour.ing · Scour
- ScourRSSBot · Scour
- [email protected] · The New York Times
- SentiBot · SentiOne
- serendeputybot · Serendeputy
- SFDC-Callout · Salseforce
- Shap-User · Parallel
- ShortPixel · ShortPixel
- Silvestro · Silverstream AI
- SimplePie · SimplePie
- Slack-ImgProxy · Slack
- Slackbot-LinkExpanding · Slack
- SlackbotLinkExpanding · Slack
- SlickBot · Slickstream
- slopsearch · Christopher Albert
- SMTBot · Similarweb
- Snap URL Preview Service · Snap
- Snapchat · Snapchat
- Sottopop · UpContent
- Sottopop-user-rss · UpContent
- SottopopNone · UpContent
- Spyglasses Bot
- StartmeBot · Start.me
- SteamChat · Valve Software
- SubstackContentFetch · Substack
- Superfeedr · Superfeedr
- SurdotlyBot · Sur.ly
- Synapse · Matrix
- Taboolabot · Taboola
- TechSpyBot · TechSpy
- TheSkinnyAI
- TikTokSpider · ByteDance
- Tiny · Tiny Tiny RSS
- TongyiBot · Alibaba
- TrendsmapResolver · Trendsmap
- TweetmemeBot · Meltwater
- TwinAgent · Twin
- Twingly · Twingly
- Twingly Recon-Sjostrom · Twingly
- uipbot · Semasio
- um-FC · Ubermetrics
- um-IC · Unicepta
- um-LN · Unicepta
- UpContent · UpContent
- UpContent-QA · UpContent
- UpContent-Test · UpContent
- Upstash QStash · Upstash
- UseAI · Use AI
- Ventus Agent
- Verity · GumGum
- Viber · Viber
- virustotal · VirusTotal
- Visually.io Shopify Editor · Visually.io
- vkShare · VK
- WAVE Streaming Platform
- weborama-fetcher · Weborama
- webspidermount · AspenTechLabs
- WebTrackrCrawler · Affsignal
- webzio · Webz.io
- WhatsApp · Meta
- WireReaderBot · Wire
- wknd-bot · Wunderkind
- WovnCrawler · WOVN
- YaDirectFetcher · Yandex
- YaK · Linkfluence
- YelpBot · Yelp
- YiyanBot · Baidu
- ZipchatBot · Zipchat Inc
- ZoteroTranslationServer · Wikimedia
AI Agent
- 2ip bot · 2IP
- AgentReadinessScanner · Cloudflare
- AutomatonHeadlessChrome · Stack Moxie
- Bugsnag script fetcher · Bugsnag
- Chirp · Binary Canary
- Claude-Code · Anthropic
- Code · GitHub
- Crazy Egg · CrazyEgg
- Criticalcss.com · Critical CSS
- Cursor · xAI
- Datadog · Datadog
- Datadog Synthetic · Datadog
- DMBrowser · Dotcom-Monitor
- Do Not Track Verifier · Electronic Frontier Foundation
- DreamHost Data Team · DreamHost
- Dubbotbot · DubBot
- EFF DNT policy checker · Electronic Frontier
- ELB-HealthChecker · Amazon
- FreshpingBot · Freshworks
- GitHub Copilot Agent · Microsoft / GitHub
- Google-Gemini-CLI · Google
- Google-Structured-Data-Testing-Tool · Google
- GoogleAssociationService · Google
- grafana · Grafana Labs
- Honeybadger Uptime Check · Honeybadger
- jetmon · Automattic
- kinsta · Kinsta
- LogicMonitor SiteMonitor · LogicMonitor
- monitoring360 · 360 Monitoring
- monitoring360bot · 360 Monitoring
- montastic-monitor · Montastic
- NewRelicbot · New Relic
- Nitro-Optimizer-Agent · NitroPack
- NIXStatsbot · NixStats
- OhDear.app · Oh Dear
- OnlineOrNot.com_bot · OnlineOrNot
- opencode · Anomaly
- parsely.com · Parse.ly
- Pingdom.com_bot · SolarWinds
- pingping.io · PingPing
- PRTGCloudBot · Paessler
- pulsetic · Pulsetic
- pulsetic.com · Pulsetic
- PWABuilderHttpAgent · Microsoft
- Rackspace Monitoring · Rackspace
- Scrunchbot · Scrunch
- SemrushBot-SWA · Semrush
- SentryUptimeBot · Sentry
- SequelWP Uptime Monitoring Bot · Nexigen Digital
- Server Density Service Monitoring · StackPath
- Silktide · Silktide
- Stape · Stape
- SwifteqLinkChecker · Swifteq
- synthetic-monitoring-agent · Grafana Labs
- t3versionsBot · T3Versions
- Testomatobot · Testomato
- Trae · ByteDance
- updown.io · Updown
- updown.io daemon · updown.io
- uptimedoctor · uptimedoctor
- UptimeStatistics · UptimeStatistics
- Validator.nu · W3C
- W3C_CSS_Validator · W3C
- W3C_Validator · W3C
- W3C-checklink · W3C
- WebsiteOps · Watchful
- websitepulse checker · WebSitePulse
- WellKnownBot · Well-Known
- WP-Rocket-SaaS · WP Rocket
- WPMU DEV Broken Link Checker Spider · WPMU DEV
- WPMU DEV Hub · WPMU DEV
- wpmudev · WPMUDEV
- WPMUDEV Uptime Monitor · WPMU DEV
- WPUmbrella · WP Umbrella
AI Search
- AI Search · Cloudflare
- AIWebIndex-Agent · Lyrenth
- Amazon Kendra · Amazon
- Amazonbot · Amazon
- Amzn-SearchBot · Amazon
- Anchor Browser · Anchor
- Anomura · Direqt
- Applebot · Apple
- Atlassian (Rovo AI) · Atlassian
- atlassian-bot · Atlassian
- AwarioSmartBot · Awario
- AzureAI-SearchBot · Microsoft
- Big Sur AI · Big Sur AI
- BorderxBot · Borderxlab (Nubestore AI)
- Brandwatch · Brandwatch
- Bravebot · Brave Software, Inc.
- CitibotSiteCrawler · Citibot
- Claude-SearchBot · Anthropic
- Cloudflare AutoRAG · Cloudflare
- Cloudflare Crawler · Cloudflare
- Cohere Training Data Crawler · Cohere
- Cotoyogi · Research Organization of Information and Systems
- crawl4ai · Open-source
- Daric2 · Cloudflare
- Daric3 · Cloudflare
- Daric4 · Cloudflare
- DeepSeekBot · DeepSeek
- Direqt Anomura · Direqt
- Echobot Bot · Echobox
- ExaSearchBot · Exa
- Factset_spyderbot · Factset
- FirecrawlAgent · Mendable / Firecrawl
- FishBot · FishBot
- Google-CloudVertexBot · Google
- Google-NotebookLM · Google
- GoogleOther · Google
- GPTBot · OpenAI
- ICC Crawler · NICT
- Kimi-SearchBot · Moonshot AI
- KimiBot · Moonshot AI
- KlaviyoAIBot · Klaviyo
- laion-huggingface-processor · LAION / Hugging Face
- LINER Bot · Liner Bot
- LinkupBot · Linkup
- Make.com · Make.com
- Meta-ExternalAgent · Meta
- meta-webindexer · Meta
- MistralAI-Index · Mistral
- netEstate Imprint Crawler · netEstate
- Novellum AI Crawl · Novellum
- OAI-SearchBot · OpenAI
- payroll-bot · ADP
- PerplexityBot · Perplexity
- PetalBot · Huawei
- QualifiedBot · Qualified.com, Inc.
- SemrushBot-OCOB · Semrush
- SemrushBotSwa · Semrush
- ShapBot · Parallel
- Spawning AI · Spawning
- TalarionSearchBot · Talarion
- TavilyBot · Tavily
- WARDBot · WEBSPARK
- xAI-SearchBot · xAI
- YandexAdditional · Yandex
- YandexAdditionalBot · Yandex
- YepBot · Ahrefs
- YGS Group Falconer Scraper
- ZanistaBot · Zanista
AI Training
AI Scraper
- AI2Bot · Ai2
- Ai2Bot-Dolma · Ai2
- Applebot-Extended · Apple
- bedrockbot · Amazon
- Bibliotheque Nacional de France Crawler · Bibliothèque nationale de France
- Bytespider · ByteDance
- CCBot · Common Crawl
- ChatGLM-Spider · Zhipu AI
- ClaudeBot · Anthropic
- CloudVertexBot · Google
- CMU-Cylab · Carnegie Mellon University
- cohere-training-data-crawler · Cohere
- CohereBot · Cohere
- Datenbank Crawler · netEstate
- Doubaobot · ByteDance
- ERNIEBot · Baidu
- FacebookBot · Meta
- Google-Extended · Google
- ICC-Crawler · NICT
- imageSpider · ByteDance
- Kangaroo Bot · Kangaroo LLM
- kb.dk_bot · Netarkivet
- KimiCrawler · Moonshot AI
- laion-huggingface-processor · LAION
- LCC · University of Leipzig
- Library Of Congress Web Archiving · United States Library of Congress
- micro-crawl · Reflection AI
- MistralBot · Mistral
- MoonshotBot · Moonshot AI
- MoonSpider · Moonshot AI
- omgili · Webz.io
- PanguBot · Huawei
- QwenBot · Alibaba
- Reflectionbot · Reflection
- SalamandraVLM · Aithlas
- SBIntuitionsBot · SB Intuitions
- Spider · Spider
- Timpibot · Timpi
- TurnitinBot · Turnitin
- VelenPublicWebCrawler · Velen
- webzio-extended · Webz.io
Archiver
- Archive-It · Internet Archive
- Arquivo Web Crawler · Arquivo
- Arquivo-web-crawler · Arquivo
- bl.uk_lddc_bot · British Library
- bne.es_bot
- bnf.fr_bot · Bibliothèque nationale de France
- CUCH.org Archive Bot · C U Cyber History
- heritrix · Internet Archive
- ia_archiver · Internet Archive
- ia_archiver-web.archive.org · Internet Archive
- IABot · Wikimedia Foundation
- Internet Archive · Internet Archive
- Internet Archive - Archive-It · Archive-It
- mirrorweb · MirrorWeb
- netarkivindsamling · Netarkivet
- NewsBank · NewsBank
- Nicecrawler · NiceCrawler
- special_archiver · Internet Archive
- Turnitin · Turnitin
- XY-Archive-Compliance · XY Archive Compliance
- XY-Archive-Compliance-Archiver · XY Planning Network
Browser automation
Monitoring
- 360Monitoring · 360Monitoring
- AccessStatus · AccessStatus
- ActiveComply Bot · ActiveComply
- Adagio Bot · Adagio
- Advailo Kuma · Advailo
- AgencyToolsPro Uptime Monitoring Bot · AgencyToolsPro
- Alertsite by Smartbear · Smartbear
- Area 360 · Area360
- Audisto Crawler · Audisto
- Automaton · Automaton
- Better Uptime · Better Stack
- BigScootsMonitor · BigScoots
- BinaryCanary · Binary Canary
- BlogVault · BlogVault
- Brynson Compliance Bot · Brynson
- BugsNag · Bugs Nag
- BuiltWith · BuiltWith
- catchpoint · Catchpoint
- Checkly Bot · Checkly
- CheckView · CheckView.io
- Cloudflare CustomerSLO Prober · Cloudflare
- Cloudflare Diagnostics · Cloudflare
- Cloudflare Observatory Curly · Cloudflare
- Cloudtrellis · Cloudtrellis
- ContentKing · ContentKing
- ContentSquare Static Resource Scraper · Contentsquare
- Crazy Egg Bot · CrazyEgg
- deadlinkchecker · DLC Websites
- Deepcrawl · Lumar
- Doctom Monitor · Dotcom-Monitor
- Dr. Link Check · Dr. Link Check
- Easydns · EasyDNS
- elmah.io Uptime Monitoring · elmah.io
- eRepublik.tools · Sebastian Foth - Software Solutions
- EvoUptimeBot · Evo Agency Ltd.
- FDL Stats Bots · FTW Entertainment LLC
- Foregenix ThreatView/WebScan · Foregenix Limited
- FullStory · Full Story
- Gatus · Gatus
- Ghost Inspector · Ghost Inspector
- Google Schema Markup Testing Tool · Google
- Grafana Synthetic Monitoring · Grafana Labs
- Grafana's Synthetic Monitoring · Grafana Labs
- GTmetrix · GTmetrix
- GuestpostsBot · Guest Posts
- HetrixTools · HetrixTools Inc
- honeybadgerbot · Honeybadger Industries
- HostTracker · HostTracker
- Hotjar · Hotjar
- InternetArchiveBot · Internet Archive
- IsDownBot · IsDownBot
- Jagged Pixel UptimeBot · Jagged Pixel Inc.
- Jetpack · Automattic
- KargoBot-Artemis · Kargo
- Kinsta · Kinsta
- Link Tiger · LinkTiger, Inc.
- logicmonitor · LogicMonitor
- Macrobondbot · Macrobondbot
- Mavifinds Bot · Mavifinds
- mediaboardbot · Mediaboard
- Medialogia Bot · Medialogia
- Monitage · Monitage
- Monitis · Monitis
- Monsidobot · Acquia
- MonSpark · MonSpark
- MontasticMonitor · Montastic
- Netumo · Netumo
- NETVIGIE · Netvigie
- New Relic · New Relic
- NixStatsMonitoringBot · NixStats
- Nodeping · Nodeping
- Oh Dear · Oh Dear
- Onetrust CMP Scanner · OneTrust, LLC.
- OnlineOrNot Monitor · OnlineOrNot
- OutsellURLValidator · Outsell Corporation
- PaesslerCloudBot · Paessler
- Panopta · Panopta
- Parse.ly · Parse.ly
- Pingdom · Pingdom
- PingPing · PingPing
- Pulsetic · Pulsetic
- Qualys SSL Scanner · Qualys
- Rackspace · Rackspace
- RetroListeCOM · Niclas Papst
- Robinbot · Robin Education
- SE Ranking Bot · SE Ranking
- Sentry Uptime Monitoring · Sentry Uptime Bot
- SequelWP · SequelWP
- Server Density · Stackpath
- SISTRIX Optimizer · SISTRIX GmbH
- Site24x7 · Site24x7
- SiteUpTimeBot · SiteUpTime
- SolarWinds Observability · SolarWinds
- Spectate · Spectate
- Splunk · Splunk
- Splunk Synthetics · Splunk
- Stape Scanner · Stape Inc
- StatusCake · StatusCake
- Swifteq Link Checker · Swfiteq Ltd
- Synthetic Bot · DataDog
- TermlyBot · Termly
- Thousand Eyes Cloud Agent · Thousand Eyes
- Trendiction Bot · Trendiction S.A.
- UpDownBot · Updown
- Uptime LLC · Uptime
- Uptime Monitoring · GoDaddy
- UptimeBot · UptimeBot
- UptimeRobot · Uptimerobot
- Uptimia · Uptimia
- UptrendsBot · Uptrends GmbH
- Watchbot · WatchBot
- Watchful · Watchful LLC
- webpagetest · WebPageTest
- WebsitePulseBot · WebsitePulse
- Webstatus247 · WebStatus247
- WkndHealthCheckBot · Wunderkind
- WormlyBot · Wormly
- WP Umbrella · WP Umbrella
- WPMU DEV Broken Link Checker · WPMUDEV
- WPMUDEV Uptime Monitor 5.0 · WPMUDEV
- Zvelo · zvelo
Other
- 2checkout · Verifone
- Accessible Web Bot · Accessible Web
- Adyen · Adyen
- AfterShip Magento 2 Connector · AfterShip
- Akismet · Akismet
- Amazon Bedrock AgentCore Browser · Amazon
- Apple App Site Association · Apple
- Authory · Authory
- Bitbucket · Atlassian
- Bling ERP · Bling
- ChargeBee · ChargeBee
- Cloudflare CSUP · Cloudflare
- Cloudflare Validator · Cloudflare
- Coinbase Webhooks · Coinbase
- Critical CSS Bot · Critical CSS
- cron-job.org · cron-job.org
- EasyBill Import Manager · easybill.de
- EasyCron · EasyCron
- Exodus · Exodus
- Flashy-PHP/1.0.0 · Flashy
- Freshping · Freshworks
- Google API · Google
- Google Read Aloud · Google
- GoPay · GoPay.cz
- HeadlessChrome
- HelloWork · HelloWork Group
- HIFIBot · HIFI
- HSTS preload bot · Chromium
- HubSpot Crawler · HubSpot
- Instapaper · Instant Paper
- Integromat · Make
- Known Agents Browser · Known Agents
- MainWP · Direct Support / Website Managed
- Make Platform · Make
- ManageWP · ManageWP
- MelonMesa Bot · MelonMesa
- Metorik · Metorik
- Modular DS · Uniqoders Technologies SL
- Mollie Bot · Mollie B.V.
- Montonio Webhooks · Montonio
- New York Times Newsgathering · The New York Times
- Noibu JS Beautifier · Noibu
- Notion Automation · Notion
- Omnisend · Omnisend
- Paddle Webhooks · Paddle
- PayPal · PayPal
- Polar · Polar
- PrintFriendly · PrintFriendly.com
- Pro Sitemaps · PRO Sitemaps
- QuickPay Callback · QuickPay
- Rakuten Image extraction bot · Rakuten
- Razorpay Webhook · Razorpay
- Readable · Added Bytes Ltd
- Recurly Webhooks · Recurly
- Retool · Retool
- SendGrid · SendGrid
- ServerHunterSpider · Server Hunter
- Shopify-Captain-Hook · Shopify
- SilktideBot · Silktide
- Skroutz ImageBot · Skroutz S.A.
- Slackbot · Slack
- Snipcart · Snipcart
- Sora Caisse POS · Sora Caisse POS
- Spark Shipping · Spark Shipping
- SparkpostBot · Bird
- StatistikAustria · Statistik Bot
- Stripe · Stripe
- stripe_apple_pay_verify_ownership · Stripe
- Stripebot · Stripe
- Svix Webhooks · Svix Inc.
- Trustly · Trustly Group AB
- Twilio Proxy · Twilio, Inc.
- VaultPress · Automattic
- Wayflyer · Wayflyer
- WMF Citoid · Wikimedia Foundation
- WMF Zotero Translation Server · Wikimedia Foundation
- Wordpress Namecheap · Namecheap
- Worldline Bot · Worldline
- WorldPay · WorldPay
- WOVN Crawler · Wovn Technologies, Inc.
- WP Time Capsule · WP Time Capsule
- XY Archive Compliance Bot · XY Archive Compliance
- Yokoy Group Webhooks · Yokoy Group AG
- Zapier · Zapier Inc.
- Zendesk Webhook · Zendesk
SEO
- 360Spider · Qihoo 360
- Alexa Archive · Alexandria.org
- alexa site audit · Amazon
- AlexandriaOrgBot · Alexandria.org
- Algolia · Algolia
- All Africa Crawler · AllAfrica Global Media
- Atom Feed Robot · RSSMicro
- Baidu ADS Server Proxy · Baidu
- Baiduspider · Baidu
- Baiduspider-render · Baidu
- bingbot · Microsoft
- BingBot · Microsoft
- Cloudflare Web Scanner · Cloudflare
- cludo.com bot · Cludo
- Cốc Cốc · Coccoc
- coccocbot-image · Coc Coc
- coccocbot-web · Coc Coc
- Coveo Bot · Coveo
- crawler.freespoke.com · Freespoke
- Crawlson · Crawlson
- Dataprovider · Dataprovider.com
- Dataprovider.com · Dataprovider.com
- Daum · Daum
- DocSpotBot · DocSpot
- DuckDuckBot · DuckDuckGo
- DuckDuckGo-Favicons-Bot · DuckDuckGo
- Feedfetcher-Google · Google
- FindFiles.net · FindFiles.net
- FindITAnswersbot · IT Influentials
- Freespoke · Freespoke
- FreespokeCrawler · Freespoke
- Funnelback · Squiz - FunnelBack
- fyndbot · Fynd
- FyndSearchEngine-Crawler · Fynd
- FyndSearchEngine-ReCrawler
- GeedoProductSearch · Geedo
- Gigabot · Gigablast
- Google Favicon · Google
- Google Images · Google
- Google Scholar · Google
- Google Videos · Google
- Google-Flights-Search-Spackle · Google
- google-xrawler · Google
- GoogleBot · Google
- Googlebot-IA · Google
- Googlebot-Image · Google
- Googlebot-Mobile · Google
- Googlebot-News · Google
- Googlebot-Video · Google
- Greppr Web Crawler · Greppr
- HaosouSpider · Qihoo 360
- Hype Machine · Hype Machine
- IbouBot · Babbar
- intelx.io_bot · Intelligence X
- Jambot · JamBot
- Jobiglo-Bot · Jobiglo
- jobswithgptcom-bot · JobsWithGPT
- Jooblebot · Jooble
- Kagi Bot · Kagi
- Kagibot · Kagi
- Level9SearchBot · Level 9
- Linespider · LINE Corporation
- Luxxlebot · Luxxle
- lyonl · Lyonl
- lyonl-crawler · Lyonl
- MagiBot · Peak Labs
- Marginalia Search · Marginalia
- Mars Finder · Mars Flag
- Mojeek · Mojeek
- MRGbot · MRG Web Services srl
- MSN · Microsoft
- msnbot-media · Microsoft
- mwmbl · Mwmbl
- Neevabot · Neeva
- NestDaddyBot · NestDaddy
- NewsNow · NewsNowUK
- Notabot · Helpfeel
- Potions · Potions
- Qwantbot · Qwant
- Qwantbot-official · Qwant
- Qwantify · Qwant
- Scrubby · Scrub the Web
- search.marginalia.nu · Marginalia
- Seekport · SISTRIX
- SeekportBot · SISTRIX
- SemanticScholarBot · Ai2
- Seznam · Seznam
- SeznamBot · Seznam
- SiteSearch360 · SEMKNOX GmbH
- SMTnet PM Bot · SMTnet
- Sogou Push Spider · Sogou
- Sogou web spider · Sogou
- SokoCrawler · eXperience Tech
- stepstoneCrawlBot · StepStone
- StractBot · Stract
- Teoma · Ask
- Terracotta · Ceramic
- TinEye · TinEye
- TinEye-bot · TinEye
- TinEye-bot-live · TinEye
- TinEye-Web · TinEye
- Toutiao · Bytedance
- WepchSearchEngine · Wepch
- Y!J-BRZ/YATSHA crawler · Yahoo
- yacybot · YaCy
- Yahoo Slurp · Yahoo
- Yahoo! Slurp · Yahoo
- YandexBot · Yandex
- YandexImages · Yandex
- YandexMobileBot · Yandex
- YandexRenderResourcesBot · Yandex
- YandexVertis · Yandex
- Yeti · Naver
- Yeti by Naver · Naver
- YisouSpider · Yisou
- YouBot · You.com
- ZoomInfo · ZoomInfo
- ZumBot · Zum Internet Corp
Security
- AccessibleWebBot · Accessible Web
- Acunetix · Invicti Security
- AITrustRadar · AI Trust Radar
- AliyunSecBot · Alibaba
- Arachnid Shield API · Canadian Centre for Child Protection
- Black Duck Fast Dynamic · Black Duck
- BlackVeil-Security-Scanner · BlackVeil
- Blockaid · Blockaid
- Bushbaby · Cloudflare
- CensysInspect · Censys
- Cert Chief · Chief Tools
- CertChief · Chief
- cf-qualys-scanner · Qualys
- Cloudflare Radar URL Scanner · Cloudflare
- Cookie Hub · CookieHub
- cookie maestro · Cookie Maestro
- cookie-maestro · Cookie Maestro
- Cookiebot · Cybot A/S
- CookieHub · CookieHub
- CookieHubVerify · CookieHub
- CookieYesBot · CookieYes
- Cybaa Bot
- CyberOKInspect · CyberOK
- cypex.ai/scanning · Cypex
- Detectify · Detectify
- Drata Autopilot · Drata
- Dratabot · Drata
- Expanse · Palo Alto Networks
- FastDAST · Black Duck
- Foregenix · Foregenix
- Google Inspection Tool · Google
- Google Safety · Google
- Google Site Verification · Google
- Google Trust Services (DCV Check) · Google
- Google-BusinessLinkVerification · Google
- Google-InspectionTool · Google
- Hardenize · Hardenize
- Infrawatch · Infrawatch
- InternetMeasurement · Driftnet
- ISSCyberRiskCrawler · ISS
- Jumio · Coinbase
- K7MLWCBot · K7 Computing
- LegalMonster · Legal Monster ApS
- Let's Encrypt · Let's Encrypt
- Lets Encrypt Validation Server · Let's Encrypt
- Netcraft · Netcraft
- Netcraft SSL Server Survey · Netcraft
- Nikto · Open-source
- Nmap Scripting Engine · Nmap
- Observer · Observer
- Odin · ODIN
- OKX-dolphin-crawler · OKX
- OneTrust · OneTrust
- Probely · Probely
- Project Honeypot · Unspam Technologies, Inc
- ProjectShield Url Check · Google
- ProjectShield-UrlCheck · Google
- Qualys · Qualys
- Redactus Web Classifier · Redactus
- Reward Gateway · Reward Gateway
- Sansec Security Monitor · Sansec Security Monitor
- SecurityHeaders · Security Headers
- Sentry · Sentry
- SiteLock · SiteLock
- SiteLockSpider · SiteLock
- SmarshBot · Smarsh
- Software Security Research · Ruhr University Bochum
- Splunk Attack Analyzer · Splunk
- sqlmap · Open-source
- SSL Labs · Qualys
- Sucuri · Sucuri
- TwinWaveScanner · Splunk
- Usercentricsbot · Usercentrics
- Verispider · Project Honey Pot
- VirusTotalScan · VirusTotal
- visionheight.com/scan
- WebTotemBot · WebTotem
- Wordfence Central · Wordfence
- WPScan · WPScan LLC
- WTotem · WebTotem
- WTotemBot · WebTotem
SEO
- aa · AgencyAnalytics
- AddSearchBot · Addsearch
- AddThis · AddThis
- AdsTxtCrawler · OneTag Limited
- Agency Analytics Crawler · Agency Analytics
- Ahrefs Site Audit · Ahrefs
- AhrefsBot · Ahrefs
- Alli AI Bot · Alli AI
- Amazon AdBot · Amazon
- Amazon Contxbot · Amazon
- Attracta · Attracta
- Barkrowler · Babbar
- BestChange Bot · BestChange
- BigUpData Bot · BigUpData
- Bing Ads · Microsoft
- BLEXBot · WebMeUp
- BoardGamePrices Bot · KP Software Consult ApS
- Botify · Botify
- BrightEdge Bot · BrightEdge
- CaliberBot · Conductor
- Capital One Bot · Capital One
- Channable · Channable
- Channel3Bot · Channel3
- Chrome-Lighthouse · Google
- ClarityBot · seoClarity
- ClearscopeBot · Mushi Labs
- Clickagy · Clickagy
- Cloudflare Custom Error Page Crawler · Cloudflare
- Cloudflare Custom Hostname Verification
- Cloudflare Digicert DCV · Cloudflare
- Cloudflare Healthchecks · Cloudflare
- Cloudflare Prefetch · Cloudflare
- Cloudflare Purge · Cloudflare
- Cloudflare SpeedTest · Cloudflare
- Cloudflare SSLDetector · Cloudflare
- Cloudflare Stream Webhook · Cloudflare
- Cloudflare-Healthchecks
- Cloudflare-SSLDetector
- Cloudflare-Traffic-Manager · Cloudflare
- cludo.com · Cludo
- Cocolyzebot · Cocolyze
- cognitiveSEO Crawler · cognitive SEO Internet Marketing Tools.
- ContextualBot · Outcomes
- Convermax · Convermax
- CriteoBot · Criteo
- Cxense · Cxense
- DataForSEO · DataForSEO
- DataForSEO Bot · DataForSEO
- DataForSeoBot · DataForSEO
- dcrawl
- deepcrawl.com · Lumar
- DigiCert DCV · DigiCert
- dlvr.it Feed Fetcher · dlvr.it
- DomainStatsBot · Game Lounge
- Dragonbot · Dragon Metrics
- eMoney Advisor · eMoney Advisor
- Epivoz Crawler · Techmeme
- EzoicBot · Ezoic Inc
- Fedicabot · Fedica
- FedReporter Bot for FFIEC · Fed Reporter, Inc.
- Google AdsBot · Google
- Google Adsense: Mediapartners-Google · Google
- Google Digital Asset Links · Google
- Google StoreBot · Google
- Google-AdWords-Express · Google
- Google-Adwords-Instant · Google
- Google-Display-Ads-Bot · Google
- Google-Structured-Data-Testing · Google
- Google-TravelAds-Live · Google
- Groovinads · Groovinads
- HelloworkJobPostingBot · Hellowork
- HTTrack
- HTTrack 3.0
- Huckabuy Bot · Huckabuy
- hypestat · HypeStat
- IAS crawler · Integral Ad Science
- Idealo · Idealo internet GmbH
- IndeedJobBot · Indeed
- James BOT · cognitiveSEO
- klaviyo · Klaviyo
- KrawlerBot · Krawler
- LeikiBot · DoubleVerify
- LibreCrawl
- LINE OGP Scraper · LINE
- LinkCheck · Siteimprove
- LinkCheck by Siteimprove · Siteimprove
- LinkChecker Bot · LinkChecker
- linkchecker.pro · LinkChecker
- LinkCheckerBot · LinkChecker.pro
- linkdexbot · Linkdex
- MagnetmeBot · Magnet.me
- marketgoo · marketgoo
- MBCrawler · SEOptimer
- MediaMonitoringBot · MediaMonitoringBot
- Mediatoolkitbot · Determ
- Mediavine Medatada Parser · Mediavine
- MegaIndex.ru · MegaIndex
- Meta-ExternalAds · Meta
- MetaInspector
- MetaJobBot · METAJob
- MgidBot · MGID
- MirrorWebCrawler · MirrorWeb Ltd
- Missinglettr Bot · Missinglettr
- MJ12Bot · Majestic
- ModernizeBot · ModernizeYourWebsite
- MotoMinerBot · Motominer
- Moz dotbot · Moz
- Moz rogerbot · Moz
- newspaper
- NitroBot · NitroPack Ltd
- Noorobot · Noor Digital Agency AB
- NostoCrawlerBot · Nosto
- Nostra · Nostra
- Nutch
- nyt_scraping · The New York Times
- OAI-AdsBot · OpenAI
- Offline Explorer · MetaProducts
- OnCrawl · Oncrawl
- OnCrawl Mobile · Oncrawl
- Online Webceo Bot · Online Webceo
- OpenindexSpider · Openindex
- Outbrain · Outbrain
- ParticleNewsBot
- PinterestBot · Pinterest
- prerender · Prerender, LLC
- Pricey · Wilico
- Pro Sitemaps Generator · Pro-Sitemaps
- Productsup Crawler · Productsup
- Proximic · ComScore
- Quantcastbot · Quantcast
- Quartr Crawler · Quartr
- RankMath Link Checker · Rank Math
- RavenCrawler · Raven
- Reelevant · Reelevant
- RevvimGort · Revvim
- rogerbot-crawler · Moz
- RSiteAuditor · DataForSEO
- RyteBot · Ryte
- Salesforce · Salesforce
- SalesViewerBot · SalesViewer GmbH
- Scrapy · Scrapy
- Screaming Frog SEO Spider · Screaming Frog
- SearchAtlas Bot · Search Atlas
- SEBot-WA · SE Ranking
- Semrushbot · Semrush
- SemrushBot-BA · Semrush
- SemrushBot-CT · Semrush
- SemrushBot-SA · Semrush
- SemrushBot-SI · Semrush
- SemrushBotBacklinkAudit · Semrush
- SemrushBotBacklinks · Semrush
- SemrushBotLinkBuilding · Semrush
- SemrushBotSI · Semrush
- Senutobot · Senuto
- SEO Audit Check Bot · SEO Audit Check
- seo4ajax · Capsule Code
- seo4ajax.com · Capsule Code
- SeobilityBot · Seobility
- SEOkicks · SEOkicks
- SEOlizer · SEOLizer
- SERankingBacklinksBot · SE Ranking
- SerpMinerBot · Serp Miner
- serpstatbot · Serpstat
- SerpstatBot · Serpstat
- SiteAuditBot · Semrush
- Sitebulb · Sitebulb
- SiteCheckerBotCrawler · Sitechecker
- SiteGuruCrawler · SiteGuru
- Siteimprove Crawl · Siteimprove
- Siteliner · Siteliner
- Slickstream · Slickstream
- SmartologyBot · Smartology
- Statabot · StataCorp LLC
- StatsDroneBot · StatsDrone
- StatusNestBacklinkSpider · StatusNest
- Trellis-Services · Mediavine
- TTD Content · The Trade Desk
- upday · upday GmbH & Co. KG
- videootv Bot · Digital Green
- W3 Validator Services · World Wide Web Consortium (W3C)
- WebSpiderMount · AspenTechLabs Inc
- woorankreview · WooRank
- XoviOnpageCrawler · XOVI
- Yahoo Ad Monitoring · Yahoo
- Yahoo Japan · Yahoo Japan Corporation
- Yahoo Japan SEO Crawler · Yahoo
- Yext Inc · Yext
- Zoombot · SEOZoom
Social Media
- AppcastBot · Appcast
- Arena Bot · Arena.im
- Artemis Web Reader · capjamesg
- Bazqux · Bazqux
- Bing Preview · Microsoft
- Blogtrottr · Blogtrottr
- Bluesky Link Preview Service · Bluesky
- Buffer Link Preview Bot · Buffer
- Buttondown RSS-Feed-Parser · Buttondown
- ConvermaxBot · Convermax Corp.
- discordBot · Discord, Inc.
- Embedly · Embedly
- FacebookExternalHit · Meta
- Fastmail Bot · Fastmail
- Feedbin · Feedbin
- Feeder · Really Simple AB
- Feedly · Feedly
- FeedOtter · FeedOtter
- FeedWind Crawler · Mikle
- FIT Feeds Fetcher · follow.it
- FlipboardProxy · Flipboard
- FlipboardRSS · Flipboard
- FullFact Ingest · FullFact
- Google Calendar Importer · Google
- Google Feed Fetcher · Google
- Google Image Proxy · Google
- Google Page Renderer · Google
- Google Publisher Center · Google
- Google Web Snippet · Google
- Hatena · Hatena
- HEY Email Privacy Proxy · 37signals
- HubSpot Feed Fetcher · HubSpot
- HubSpot Page Fetcher · HubSpot
- Iframely · Iframely
- IFTTT RSS Feed Service · IFTTT RSS Feed Servuce
- Innguma Fetcher · Innguma
- Innologica · Innologica
- KakaoTalk Scrap · Kakao
- LinkedInBot · LinkedIn
- LinksIndexerBot · Links Indexer
- Loomly Bot · Loomly
- MailRUBot · Mail Russia
- Meta-WebIndexer · Meta
- Microsoft Preview · Microsoft
- MonitoRSS · MonitoRSS
- Muck Rack Scraper · Muck Rack
- NewsBlur · NewsBlur
- Nooshub · Nooshub
- Opengraph Bot · Opengraph
- OpenRSS · OpenRSS
- Orlo Link Preview · Orlo
- OurFamilyWizard · OurFamilyWizard
- Overcast · Overcast Radio
- Ozon Web Grabber · Ozon
- Pocket Casts Feed Parser · Pocket Casts
- PressEngine Bot · PressEngine
- Protopage · Protopage Ltd
- RSS API · RSS API (by Tibush GmbH)
- rss2tg_bot · Yellow Rubber Duck Consulting
- Scour RSS Bot · Scour
- Shortwave Image Fetcher · Shortwave Communications Inc.
- SkroutzBot · Skroutz S.A.
- Skype · Skype
- Slack Image Proxy · Slack
- Steam Chat · Valve Software
- Telegram Bot · Telegram
- Tumblr · Automattic
- twitterbot · Twitter
- WordCountBot · Weglot
- Yahoo Link Preview · Yahoo
- Yahoo Mail · Yahoo
- YahooCacheSystem · Yahoo
- YahooMailProxy · Yahoo