Web scraping is reading a page and keeping only what matters
A program fetches a page, interprets it, and turns the part it needs into a row in a table. Everything difficult about it follows from the fact that the page was designed for a person.
The four questions that decide how hard a scrape will be
Difficulty is almost never about the parsing. It is about these four, in roughly this order.
Does the content exist before JavaScript runs?
If the server returns the data in the initial HTML, an HTTP client is enough and the job is minutes. If the page assembles itself in the client, you need a browser engine and the cost rises by an order of magnitude. Check the raw response before assuming.
Is it behind a login?
Authenticated content changes the problem entirely. You are no longer reading a public page but operating an account, with whatever terms that account is bound by, and session lifetime becomes the thing that determines whether the job runs at all.
How often does the layout change?
A selector is a bet that a structure will hold. On a well-maintained site that bet lasts months; on a site under active redesign it lasts days. This, not volume, is what makes scrapers expensive to own.
What did you agree to?
Terms of service, robots directives and the nature of the data itself all constrain what is defensible. A page being publicly reachable settles none of these questions on its own.
A method that survives contact with real sites
In this order, because each step can make the next unnecessary.
-
Look for the data before you look for the DOM
Open the network panel and see what the page itself calls. A documented API, an RSS feed or the JSON endpoint behind the table is faster, more stable and less contentious than any parsing you could write.
-
Read robots.txt and the terms, and record what they said
Not as a formality. The record of what you checked and when is what turns a judgement call into a defensible one, and it costs a minute.
-
Fetch with the cheapest client that works
Try a plain HTTP request first. Escalate to a browser only when you have confirmed the content is not in the response, because the browser is the expensive part of every scraper.
-
Extract against something stable
Prefer accessible names, data attributes and text anchors over generated class names and positional paths. A selector built on a hashed class name is a scheduled outage.
-
Validate before you store
Assert shape and plausibility at the boundary. A scraper that silently writes empty strings for a week is worse than one that fails loudly on day one.
-
Rate limit yourself on purpose
Match a plausible human pace and stay well inside it. Restraint is both the courteous choice and the one that keeps you unblocked.
Common questions
Is web scraping legal?
There is no single answer, because at least four bodies of law apply and they point in different directions: contract, copyright and database rights, data protection where the data identifies people, and computer misuse where access controls are involved. Publicly reachable is not the same as freely usable, and the answer changes by jurisdiction.
What is the difference between scraping and crawling?
Crawling discovers pages by following links; scraping extracts fields from a page once you have it. Most real systems do both, but they fail differently and are worth separating in your own code.
Do I need proxies?
Usually the honest answer is that you need a lower request rate. Proxy rotation treats being blocked as a network problem when it is normally a behaviour problem, and it turns an engineering question into a procurement one.
How do I keep a scraper working?
Monitor the shape of the output, not the exit code. Field fill rates and row counts detect a silent layout change days before anyone notices the numbers are wrong; a process that exits zero tells you nothing.
When the page needs a real browser
EU-hosted sessions, no proxy rotation and no evasion features. Five browser hours to see whether that fits your job.