Invalid traffic detection

Invalid traffic is automated requests counted as human visits in your analytics. When you report sessions, pageviews, or ad impressions to an advertiser, a client, or a board, invalid traffic inflates every figure. The question is not whether automated traffic reaches your site — it does — but what share of your reported traffic it represents, and whether you can defend that share under scrutiny.

Most analytics platforms, including client-side tag-based tools, filter bots by checking the request user agent against a published list. A user agent is a text string the client sends to identify itself. This approach has two structural weaknesses.

First, the list is finite. New crawlers appear faster than lists are updated. Any automated agent not yet on the list passes through and is recorded as a human session.

Second, the check is passive. It accepts whatever the client claims to be. A crawler that sets its user agent to a common browser string bypasses the filter entirely. Misrepresentation costs the operator nothing.

The result is systematic under-reporting. The bots your analytics tool catches are the well-behaved ones that did not need catching. The ones that distort your numbers are the ones that chose not to identify themselves.

Crawlers that identify themselves are transparent about their purpose. Googlebot, for example, announces itself and can be verified by reverse DNS lookup. These crawlers are legitimate and expected. Counting them as human traffic is a measurement error, not a threat.

Crawlers that do not identify themselves present a different problem. They may be scrapers harvesting content, browser automation tools running synthetic tests, or AI agents collecting training data. Because they present as browsers, client-side analytics records them as users.

Botscope's catalogue currently covers 1,953 automated agents. The breakdown by category is in the table below. Of those, 37 can be verified through reverse and forward DNS, 48 against published address ranges, and 51 by a signed token. The rest are classified by behavioural and network signals at the request level, and labelled as such.

A defensible invalid traffic figure requires a verdict for every request, not an extrapolation from a sample. Per-request classification is the practice of evaluating each incoming HTTP request against a set of signals before any analytics event fires.

Botscope runs this classification server-side. Server-side means the check happens on infrastructure you control, before the request reaches your analytics pipeline. The verdict — human, known bot, or unverified automated agent — is attached to the request record.

Because every request carries a verdict, you can calculate the share of invalid traffic as a precise ratio: classified non-human requests divided by total requests. That ratio is reproducible. You can show the input data, the classification logic, and the output. A number you can show workings for is a number you can defend.

Botscope names 774 distinct operators across the 1,186 agents it has researched that far, out of 1,953 in the catalogue. When a request matches a known agent, Botscope records the operator name, not just a generic 'bot' label. That level of attribution lets you answer follow-up questions: which automated agents account for most of the non-human share, and what do they do.

Once you have a per-request invalid traffic share, you have three practical options.

First, adjust the figures you report. Subtract verified non-human sessions from the totals before presenting them. Document the methodology so the adjustment can be audited.

Second, segment the non-human traffic. Not all automated requests are equally harmful to your numbers. A Googlebot crawl inflates raw request counts but does not affect session-based metrics if filtered correctly. An unidentified browser automation agent that fires your analytics tag does affect session counts. Treat these differently.

Third, consider enforcement selectively. Botscope does not block traffic by default. You turn enforcement on per agent category once you have reviewed the data and confirmed you understand the consequences. Blocking a crawler you did not recognise as a legitimate partner, for example, is a risk you can avoid by classifying before acting.

Botscope does not hold ad-fraud certification or IAB accreditation. If your advertiser contract or campaign audit requires a certified measurement vendor for ad impression validation, Botscope is not a substitute for that requirement.

Botscope classifies 1,953 agents as of the current catalogue. Automated agents outside the catalogue are flagged as unverified rather than silently passed through, but they do not receive an operator-level attribution until they are added to the catalogue.

Classification confidence varies by agent type. The 1 agent in the catalogue that uses a signed token can be verified with high certainty. Agents that misrepresent themselves as browsers are classified by indirect signals, and a small proportion of those classifications will carry lower confidence scores. Botscope surfaces the confidence score per request so you can apply your own threshold when deciding what to include in a reported figure.