Crawling finds the pages; scraping reads them
A crawler starts from a seed, follows links, and keeps a record of what it has already seen. It is a graph traversal with a politeness budget, and almost every difficulty comes from one of those two words.
What a crawler is really made of
Four components, and the interesting engineering lives in the first two.
The frontier
The queue of what to fetch next. Its ordering decides whether you cover a site broadly or dive into one branch, and its deduplication decides whether you crawl the same page under fifteen URLs. Session identifiers and tracking parameters make many URLs that are one page.
The politeness budget
How fast you are willing to hit one host. A crawler without per-host rate limiting is indistinguishable from a denial of service, and the fact that it was unintentional is not a defence anyone finds interesting.
The exclusion rules
robots.txt, crawl delay, meta directives. These are conventions rather than enforcement, which is exactly why honouring them is a statement about how you operate rather than a technical constraint you had no choice about.
The store
What you keep and for how long. Crawls produce a great deal of data whose value decays quickly, and retention decided in advance is much cheaper than retention decided after the disk fills.
Crawling a site without becoming a problem
The order matters: the cheapest constraints go first.
-
Check whether a sitemap exists
A sitemap is the site telling you its own URL list. Using it is faster than discovery, less load on the host, and less likely to miss pages that are linked from nowhere.
-
Fetch and honour robots.txt per host
Per host, not per crawl. Subdomains have their own file, and caching one host's rules across a domain is a common and consequential bug.
-
Normalise URLs before queueing
Strip fragments, sort query parameters, drop known tracking keys, resolve relative paths. Deduplication that happens after the fetch has already cost you the fetch.
-
Rate limit per host and per IP
One global limit protects nobody: a hundred requests a second spread across two hosts is fifty each. The limit that matters is the one the host experiences.
-
Set an identifiable user agent
Say what you are and give a contact. An operator who can reach you will usually ask you to slow down before they block you, which is a strictly better outcome for both sides.
-
Bound depth and total pages before you start
Every crawl of an unbounded site is unbounded. Deciding the stopping condition in advance is the difference between a crawl and an incident.
Common questions
Do I need a browser to crawl?
For discovery, usually not. Links live in the HTML and an HTTP client finds them. You need a browser when navigation itself is client-side, which is common in single-page applications where the link graph only exists after the router runs.
Is robots.txt legally binding?
It is a convention, not a statute, and its legal weight varies by jurisdiction and by how the site's terms reference it. Treating it as binding is nevertheless the sensible default: ignoring it is the single clearest evidence of bad faith anyone could produce about your operation.
How do I avoid crawling the same page twice?
Normalise aggressively and hash the normalised URL, then also hash the response body. The second check catches the case where different URLs genuinely serve identical content, which normalisation alone never will.
What crawl rate is reasonable?
Slower than you think. A useful rule is one request per second per host as a ceiling rather than a target, dropped further if the site is small or response times rise. Rising latency is the host telling you something.
Crawls that identify themselves
No proxy rotation, no fingerprint spoofing. If your crawl needs to hide, we are the wrong tool and will say so.