When there is no API: extracting from closed systems

There is an unspoken assumption in most integration advice: that if you need data from a system, that system has a way to give it to you. An API, an export, a nightly file drop, something.

For a large share of the systems that matter inside real companies, this is not true. The supplier portal has a table and no export. The bank has statements you can look at. The government filing site has a submission form and a confirmation page. The ancient internal application has a screen, and the vendor was acquired in 2011.

The industry name for getting data out of these anyway is screen scraping, and it has an unfairly poor reputation — partly earned by a generation of brittle terminal-emulator tooling, and partly because it sounds like something you would rather not admit to. But the underlying situation has not changed and is not going to: the interface built for a person is frequently the only interface there is. What has changed is that the screen is now a web page, which makes it far more tractable than it used to be.

Establish that you need this at all

Half the extraction projects that get started should have been an afternoon of reading someone's documentation. Before writing anything, work through these in order, because each one can make the rest unnecessary.

Is there an undocumented API? Open the browser's network panel and use the application normally. Most modern portals are single-page applications talking to a JSON backend, and that backend is right there. Calling it directly is faster, more stable and far less likely to break than parsing the rendering of its output. It may be undocumented and unsupported, which is a real risk to weigh — but weigh it, rather than skipping past it.

Is there an export nobody mentioned? Buried in a settings page, available on request, or present for a different user role than the one you were given. Ask the vendor directly. The number of extraction projects that could have been a support ticket is not small.

Is there a scheduled report? Many enterprise systems will email a CSV nightly if someone ticks a box. Less elegant than an API, dramatically more stable than scraping, and it exists more often than people check.

Is there a partner or regulatory channel? In several regulated domains a standardised access route exists because it was mandated. Financial data access rules in Europe are the clearest example, and going through the sanctioned channel is better on every axis than working around it.

Only when all four are genuinely no does the browser become the integration layer. And at that point it is worth writing down why, because in two years somebody will ask whether this was necessary.

Four sources, four different jobs

"Extraction" covers work that shares a name and little else, and choosing the wrong technique for the source is the most expensive early mistake. The four cases, with what actually goes wrong in each:

A table in a portal. The structure is real but implicit — encoded in layout rather than in a schema. A browser reads it reliably. The failure mode is that layout is not a contract: a column is inserted, a header is renamed, pagination changes from twenty rows to fifty, and nothing announces it. This is the case the method above is written for.

A document. PDFs, scans, spreadsheets attached to emails. Position on the page carries meaning that no markup records, which is why this has its own tooling and its own specific miseries — tables that span page breaks, multi-column layouts read in the wrong order, numbers that are images. Do not try to solve this with a browser; it is a different problem with different tools.

A file that arrives. An SFTP drop, an email attachment, a shared folder. Genuinely the easiest case and frequently overlooked because it feels less modern than an API. If someone will send you a nightly CSV, take the CSV.

Free text. Notes, descriptions, correspondence. There is no structure to find, only structure to impose. This is where language models made a real difference and also where validation matters most, because a model asked to extract a field will produce a plausible one whether or not the information was present.

The trap is a pipeline that treats all four the same because they all end in a database row. They have different accuracy profiles, different failure modes and different costs, and a single "extraction service" that handles all four uniformly will be over-engineered for the file and under-engineered for the free text.

The method

Read the terms before you read the DOM

Not a formality, and not the same question as "is this technically possible".

For an authenticated system, the account terms govern. If they prohibit automated access, that is the answer, and the useful move is to go and negotiate rather than to route around it — for a supplier portal or a customer's own system, this is frequently a conversation that succeeds. Vendors say yes to a stated purpose surprisingly often when the alternative is their customer leaving.

For anything public, robots directives and the terms of use both apply, and where the data identifies people, data protection law applies on top and independently of both. Public reachability settles none of these.

Record what you checked and when. That record costs a minute and it is what turns a judgement call into a defensible one.

Model the record before you write the extractor

Name every field, its type, whether it is mandatory, and what a valid value looks like. Do this first, because it converts extraction from an open-ended activity into a checkable one and it gives validation something to check against.

It also surfaces the questions that otherwise appear at the worst moment. What does an empty value mean? Is it zero, or unknown, or not applicable? Which combination of fields identifies a record uniquely? What is the natural key you will use to detect that you have seen this before?

Extract against something that will hold

Selector stability is the entire maintenance cost of the system you are about to build. Choose accordingly.

// Worst: positional. Breaks when anyone adds a column.
const total = await page.locator('table tr:nth-child(3) td:nth-child(5)').innerText();

// Bad: generated class names. Breaks on the next build.
const total = await page.locator('.sc-kAyceB.hXBnpQ').innerText();

// Better: semantic, and it is what a screen reader uses.
const total = await page.getByRole('row', { name: /invoice total/i })
                        .getByRole('cell').last().innerText();

// Best, where it exists: the site's own stable hooks.
const total = await page.locator('[data-field="invoice-total"]').innerText();

The ordering here is not aesthetic. Accessibility roles and names are the most stable thing on most pages, because they are load-bearing for users who need them and therefore less likely to be casually regenerated by a build tool. Positional selectors encode a layout, and layouts are the thing that changes.

Where the structure is genuinely unstable — a portal that redesigns without notice, or a hundred portals that are each different — this is where a model earns its cost. But use it as a fallback rather than a default, because a selector that misses fails loudly and a model that misreads fails quietly.

Validate at the boundary

Everything extracted is untrusted until it has passed a check. The checks that catch the most are cross-field, not per-field:

function validate(invoice) {
  const errors = [];
  if (!/^[A-Z]{2,4}-\d{4,8}$/.test(invoice.number)) errors.push('number format');
  if (!(invoice.date instanceof Date) || isNaN(invoice.date)) errors.push('date unparseable');
  if (invoice.lines.length === 0) errors.push('no line items');

  // The check that finds real problems: does the document agree with itself?
  const computed = invoice.lines.reduce((t, l) => t + l.qty * l.unitPrice, 0);
  if (Math.abs(computed - invoice.net) > 1) errors.push('lines do not sum to net');
  if (Math.abs(invoice.net + invoice.vat - invoice.gross) > 1) errors.push('vat does not reconcile');

  return errors;
}

A record that fails goes to a review queue. It does not go into the system of record with a shrug, and it is not retried on the assumption that the second attempt will be luckier — a reconciliation failure is a fact about the document, not about the network.

Store the evidence with the value

For each extracted field: where it came from, when, and ideally the raw fragment. This is a small amount of storage and it is the entire difference between answering a question about a number in six months and being unable to.

It is also, incidentally, most of what a regulated process needs anyway, so the work is not wasted even where nobody has asked for it.

Monitor the shape, not the exit code

The characteristic failure of an extraction pipeline is not a crash. It is a field that quietly drops from 98 per cent populated to 60 per cent because a column moved, and it runs like that for three weeks.

Track fill rate per field and row count per run, and alert on the change rather than the absolute value. This single practice catches more real problems than everything else on this page combined, because it is the only one that detects the failures that do not raise.

Authenticated extraction is a different problem

Almost everything valuable is behind a login, and this is where extraction projects actually fail. Not on parsing — on staying logged in.

Session lifetime sets the shape of everything. If the portal expires sessions after twenty minutes, a job that takes twenty-five minutes has a bug that has not happened yet. Measure the real lifetime before designing the job, and design the job to be resumable across a re-authentication rather than to fit inside a window.

Multi-factor prompts are the norm, not the exception. Time-based codes can be handled programmatically. Push approvals, device confirmations and photo-based schemes cannot, and the honest design is a defined handover to a person rather than a heroic attempt at automation. A workflow where a human approves a prompt and the automation continues is a normal, robust pattern; one that pretends the prompt will not appear is not.

Credentials should not reach the model. If an agent is involved, the login should happen in a layer the model drives but does not observe. Otherwise the password is in the context window and, on the next turn, in whatever the page representation captured. This is worth verifying rather than assuming, in any platform including ours.

A persistent profile is worth more than it looks. A browser that arrives with the previous session's cookies and storage skips the login entirely much of the time. That is not only faster — it removes the step that fails most often, and it looks to the portal like a returning user because it is one.

What it costs to own one

Extraction systems are cheap to build and expensive to keep, and the ratio is worth understanding before committing, because it is the opposite of most software.

The build is genuinely quick. A competent engineer can get a working extractor against a single portal in a day or two, and this is what makes the first project look easy and the tenth one look like a department.

The ownership cost has three components, none of which appears in the estimate:

Breakage. Every portal you extract from is a portal someone else will redesign. The rate varies enormously — a stable enterprise system might go two years untouched, a consumer-facing site might change monthly — but across a portfolio the expected number of breakages per month scales linearly with the number of sources. Ten sources is a background task. A hundred sources is somebody's job.

Credential churn. Passwords rotate, accounts get disabled, someone enables a new multi-factor policy, a portal decides to force a re-enrolment. This scales with sources too, and it is more disruptive than breakage because it usually stops everything from that source at once rather than degrading one field.

Silent-failure surveillance. Somebody has to watch fill rates and act on them. Automatable, but only if it was built in from the start; retrofitting monitoring onto twenty extractors that were each written as a script is a project in its own right.

A useful planning heuristic: the second extractor should cost a tenth of the first, or you have built a script instead of a system. If each new source is a fresh engagement with the same problems — auth, session handling, validation shape, monitoring — the shared machinery has not been factored out and the portfolio will not scale past about a dozen sources before it stalls.

Where this sits legally, briefly

Not legal advice, and jurisdiction-dependent. But the four questions worth having answers to before you build:

What did you agree to? Terms of use for public access, account terms for authenticated access. This is contract, and it is usually the most directly binding of the four.

Does the data identify people? If so, data protection law applies to your collection of it, independently of whether the collection was technically permitted. In Europe that means a lawful basis, a purpose, and a retention period, and it applies to data that was published as much as to data that was not.

Is the source a protected database? European database rights can protect a substantial extraction from a collection even where no individual item is protected. Volume matters here in a way it does not elsewhere.

Did you circumvent an access control? This is the line that changes the character of the question in most jurisdictions. Reading a page you are allowed to see is different from defeating a measure designed to stop you, and it is a difference worth staying well clear of.

Our own position on the last one is a product decision as much as a legal one: no proxy rotation, no fingerprint spoofing, no CAPTCHA solving. If a job only works while hidden, we are the wrong tool. If it is a portal your company has an account with and a right to use, that is precisely the case we built for — and it is what the sessions are designed around.

The shape of a good extraction system

Five properties, and none of them is about parsing:

It tries the cheapest method first and escalates only when the cheap one is confirmed not to work. It fails loudly rather than writing plausible nonsense. It carries its evidence, so any number can be traced to a source and a time. It watches its own output shape, so silent degradation surfaces in days rather than months. And it stays inside what was agreed, so nobody has to relitigate the decision later.

Screen scraping got its reputation from systems that had none of these. The technique was never the problem.