Bot detection is a probability estimate, not a test
No site can prove a visitor is a program. It scores signals, sets a threshold, and accepts that both kinds of mistake will happen. Understanding that is most of understanding why you were blocked.
The four families of signal
Network reputation
The address you arrive from carries history. Datacentre ranges score differently from consumer connections, and a range that other people abused yesterday is a range you are judged by today. This is usually the largest single term and the one you influence least.
Client characteristics
Headers, TLS handshake shape, and the properties a page can read from the browser. These describe what kind of client you are. Consistency matters more than any single value: a combination that could not occur on a real device is a stronger signal than any one attribute.
Behaviour over time
Request rate, navigation order, dwell time, whether resources are fetched. This is where most automation is actually identified, and it is the family a well-behaved client naturally scores well on because it is largely a description of restraint.
Declared identity
The newest family, and the one moving fastest. Web Bot Auth and comparable schemes let a client cryptographically sign that it is a specific known agent, which turns the question from detection into verification.
What a block usually means
| Likely cause | The useful response | |
|---|---|---|
| Immediate block, first request | Network reputation | Change egress, or ask for an allowlist |
| Works, then blocks after minutes | Request rate | Slow down; this is the common case |
| Interstitial challenge on every visit | Client characteristics | Use a real browser, not an HTTP client |
| Blocked only on certain pages | Page-level policy | Check robots directives and terms |
| Blocked after login | Account-level terms | Read what the account agreed to |
| Intermittent, no pattern | Threshold near your score | Reduce rate and stabilise the client |
As of 2026-08-31 · Competitor details come from each vendor's published pricing page on that date.
Diagnosing a block without guessing
This is a reliability investigation. Treat it as one.
-
Read the response, not just the status
Many blocks carry an explanation, a reference id or a contact route in the body. Teams routinely retry for days against a page that says why in the first paragraph.
-
Establish whether it is you or your address
Try the same request from a different network by hand. If a browser on a phone succeeds, the signal is network reputation and no amount of client work will help.
-
Halve the rate and re-measure
Rate is the cheapest variable to change and the most common cause. If halving fixes it, you have found the answer and it was never about detection sophistication.
-
Check what you agreed to
Robots directives, terms, and for authenticated access the account terms. If the answer is that this access is not permitted, the engineering question is settled.
-
Ask
An identifiable user agent with a contact address gets asked to slow down far more often than it gets blocked. Many operators will allowlist a stated purpose that they would otherwise refuse.
Common questions
Is bot detection the same as CAPTCHA?
No. A CAPTCHA is one possible response to a score, not the detection itself. Most detection is invisible and most decisions are made before any challenge is shown, which is why a system that only reacts to visible challenges is reacting to the tail of the process.
Why does my automation get blocked when a browser does not?
Usually rate and consistency rather than any single tell. A person generates a handful of requests with irregular gaps and fetches every resource; automation often generates hundreds with even gaps and fetches only the document. The second pattern is visible without any fingerprinting at all.
What is Web Bot Auth?
A scheme for a client to prove which agent it is by signing requests with a key tied to a published identity, so a site can allow a known agent deliberately rather than infer intent from behaviour. Several large networks now support declaring agents this way, and it is the direction the problem is being solved in.
Do you help bypass bot detection?
No. We do not sell proxy rotation, fingerprint spoofing or CAPTCHA solving, and if your target is an adversarial public site then several other vendors fit your problem better than we do. Our position is that automation should be able to say what it is.
Automation that identifies itself
If your workload only works while hidden, we are the wrong tool, and we would rather say so before you integrate.