Updated 16 September 2026
01 · why the obvious checks stopped working
Why the obvious checks stopped working
The first generation of headless detection read navigator.webdriver, looked for a missing plugins array, or checked whether the user agent contained the word HeadlessChrome. All three were true of Puppeteer in 2018 and none of them is reliably true now.
That is not a gap somebody forgot to close. Automation frameworks are maintained by the same communities that maintain browsers, they are used for legitimate testing far more often than for scraping, and removing a fingerprinting surface is a feature request rather than an attack. A detection method built on a property the framework can simply stop setting has a shelf life measured in releases.
The practical consequence is that a check which was accurate when it was written degrades quietly. It keeps returning an answer, the answer keeps being negative, and nothing about the output says the check has stopped meaning anything.
02 · what is left to check
What is left to check
- 1 The connection. A TLS handshake has an order and a cipher list that come from the HTTP library, not from the user agent string typed above it. A client claiming a current Chrome while handshaking like a Go program is describing two different things.
- 2 The network the request came from. A browser on somebody’s desk is on a consumer network; a browser in a container is usually on a datacenter one. Neither is proof, and the second is increasingly hidden behind residential proxies, which is why it is a contributing signal rather than a verdict.
- 3 What the page reported back. A real browser runs the page’s JavaScript and answers; an automated one often does too, but the two disagree about timing, about input events that were never generated, and about whether anything was ever scrolled.
- 4 Whether the same client keeps arriving without ever loading what a page references. A browser that requests HTML and none of the assets beside it is not rendering the page it asked for.
03 · automated browsers the catalogue names
Automated browsers the catalogue names
| Bot | Operator | Default policy | Verifiable |
|---|---|---|---|
| GoogleAgent-Mariner | Google DeepMind | Observe | By user agent |
| Manus-User | Butterfly Effect | Observe | By user agent |
| AmazonBuyForMe | Amazon | Observe | By user agent |
| anthropic-ai | Anthropic | Observe | By user agent |
| Claude-Web | Anthropic | Observe | By user agent |
| cohere-ai | Cohere | Observe | By user agent |
| iAskBot | iAsk | Observe | By user agent |
| iaskspider | Ai Search | Observe | By user agent |
| KunatoCrawler | Zzazz | Observe | By user agent |
| NovaAct | Amazon | Observe | By user agent |
| WRTNBot | Wrtn | Observe | By user agent |
| ZenphonyBot | Zenphony | Observe | By user agent |
Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.
04 · what this does not do
What this does not do
None of the above identifies a headless browser with certainty, and a page that claimed otherwise would be describing a product that does not exist. A well-operated automation setup on a residential proxy, driving a patched browser at human speed, is genuinely hard to separate from a person.
What changes is the cost. Each signal that has to be defeated adds work and adds expense, and the point at which scraping stops being worth it arrives well before the point at which it becomes impossible.
The second thing that changes is that the decision is legible. When a request is stopped, the check that stopped it is recorded against that request, which is what makes a disputed block something you can look up rather than argue about.
Questions
Does this block Playwright and Puppeteer used for our own testing?
Only if you configure it to. Test traffic normally arrives from known addresses, and an address allowlist is the simplest exemption. Test runs can also carry a header the site recognises.
Is a headless browser always unwanted?
No. Uptime monitoring, accessibility auditing, link checking and previewing all use one legitimately. Which of them you want is a policy decision, which is why the catalogue records what each named agent is for rather than deciding for you.
What happens to a real visitor who trips a signal?
Under enforcement they are asked to complete a check rather than refused outright, and the decision is recorded with the signal that caused it. In observe mode nothing is enforced and the classification is still recorded.
Can a stealth plugin defeat this?
Stealth plugins remove properties the page can read in JavaScript. They do not change the TLS handshake the connection was opened with, and they cannot make a datacenter address look like a domestic one on their own.
Do I have to enforce anything to see this?
No. Observation mode classifies every request and enforces nothing, which is the state a new site starts in.
See which of these reach your site
Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.