Updated 16 September 2026
01 · why address reputation stopped being enough
Why address reputation stopped being enough
Blocking datacenter ranges used to be most of bot mitigation, and for a while it worked, because running automation meant running it on a server and servers live in identifiable address space.
Residential proxy services ended that. They pay for access to consumer connections — sometimes knowingly through a rewards app, sometimes through software the user did not read carefully — and resell them by the gigabyte. Traffic through one arrives from a genuine domestic line belonging to a genuine subscriber, geographically plausible and shared with real household traffic.
So the address now answers a narrower question than it used to. It still identifies a server when one is being used directly, and that remains a large share of automated traffic because it is far cheaper. What it cannot do is clear a request: an address that looks ordinary is no longer evidence that the client is.
02 · what an address can still establish
What an address can still establish
- 1 That the connection originates in hosting rather than consumer address space, which for a request claiming to be a person is a contradiction worth acting on.
- 2 That it is a published Tor exit node. The list is public and exact, so this is identification rather than inference.
- 3 That the operator it claims to be publishes its addresses and this is not one of them. A crawler naming itself Googlebot from outside Google’s ranges is not a probability, it is a mismatch.
- 4 That many requests share a network origin while presenting as unrelated visitors — the shape a proxy pool makes even when each individual address looks ordinary.
03 · agents whose operator publishes the addresses they use
Agents whose operator publishes the addresses they use
| Bot | Operator | Default policy | Verifiable |
|---|---|---|---|
| 360Spider | Qihoo 360 | Allow | By user agent |
| Alexa Archive | Alexandria.org | Allow | By user agent |
| alexa site audit | Amazon | Allow | By user agent |
| AlexandriaOrgBot | Alexandria.org | Allow | By user agent |
| Algolia | Algolia | Allow | By user agent |
| All Africa Crawler | AllAfrica Global Media | Allow | By user agent |
| Atom Feed Robot | RSSMicro | Allow | By user agent |
| Baidu ADS Server Proxy | Baidu | Allow | By user agent |
| Baiduspider | Baidu | Allow | By user agent |
| Baiduspider-render | Baidu | Allow | By user agent |
| bingbot | Microsoft | Allow | By user agent |
| BingBot | Microsoft | Allow | By user agent |
Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.
04 · what this does not do
What this does not do
There is no check that reliably identifies a residential proxy from the address alone, and a page claiming one would be selling something that does not work. Reputation feeds catch pools that have been observed before, which is a moving subset rather than a boundary.
Blocking every datacenter address is also a decision with a cost, not a free win. Corporate VPNs, privacy services people pay for deliberately, and the monitoring you run on your own site all arrive from hosting ranges. So does a share of mobile traffic, depending on the carrier.
This is why the address is one input here rather than the rule. Combined with a handshake that disagrees with the user agent, or with a claimed identity that fails verification, it becomes decisive; on its own it mostly describes where a request was made from.
Questions
Should I block all datacenter traffic?
Rarely. Corporate VPNs, paid privacy services and your own monitoring all come from hosting ranges, and so does some mobile traffic. It is a strong contributing signal and a blunt rule.
Can you detect residential proxies?
Pools that have been observed and listed, yes, through reputation feeds. A freshly rented consumer line that has done nothing yet is indistinguishable from a household by address alone, which is why the other signals carry the decision.
What about Tor?
Exit nodes are published, so identification is exact. Whether to treat Tor differently is a policy choice with legitimate users on both sides of it.
Do you use a commercial IP intelligence database?
Address classification and reputation feeds are used as contributing signals, alongside the ranges operators publish about themselves. Where an operator publishes its own ranges, that is preferred: it is authoritative rather than inferred.
How do I tell a VPN customer from a scraper?
Not by the address. Whether the client’s handshake matches its user agent, whether the page it asked for was actually rendered, and whether one origin is behaving like many visitors are what separate them.
See which of these reach your site
Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.