Bank-statement parsing and income verification at production accuracy

You can trust a bank-statement income figure only when three things hold: the extraction survives the messy PDFs customers actually upload, every derived number reconciles back to the transactions it came from, and you measure accuracy on a held-out set that mirrors your real traffic instead of the clean demo files. Skip one and the figure is a guess wearing a decimal point.

Statement parsing is one of those problems that looks solved for about an afternoon. You wire up an extractor, feed it four PDF statements from two banks, watch it pull transactions and sum the salary credits, and the number matches. Then you point it at a month of real applications and the accuracy you thought you had evaporates. The demo set was the easy 10 percent. The other 90 is where the work is.

The document is the hard part, not the model

Most of the failure in income verification happens before any income logic runs. It happens at extraction, because “a bank statement” is not one format.

Real inbound traffic includes native digital PDFs with a clean text layer, scans of printouts, phone photos of scans, mobile-app screen exports, CSV downloads, and the occasional password-protected file where the password is the customer’s date of birth in a format nobody documented. Each bank lays out columns differently. Some put debits and credits in one signed column, some in two. Balance carries forward across pages, and the running balance is your best friend for validation if you can read it.

So the first decision is routing. Detect whether a document has a usable text layer and take the structured path when it does; fall back to statement OCR only when you must, because OCR introduces character-level errors precisely in the digits you care about. A 1 misread as a 7 changes the income figure a loan gets underwritten against.

Aim the extractor at a structured ledger, and hold each field to that schema:

• Transaction date, posting date, description, amount, signed direction, and running balance
• Account holder identity and account number, for entity resolution across multiple statements
• Statement period boundaries, so you know the window you are measuring over
• Opening and closing balance, which anchor the reconciliation

That last point is the cheapest quality check you will ever build. Sum the transactions in order, apply them to the opening balance, and confirm you land on the stated closing balance. When it does not tie out, you have missed a transaction, misread an amount, or dropped a page. Reject the document into a review queue rather than passing a ledger you know is broken. Extraction without this reconciliation is extraction you cannot defend.

Turning transactions into an income figure

Once you have a clean ledger, income detection is a classification and aggregation problem, and this is where teams either build something honest or something that flatters itself.

The naive approach sums everything that looks like a credit. That is wrong on contact. Inbound transfers between the customer’s own accounts, loan disbursements, refunds, reversals, and one-off gifts are all credits, and none of them are income. Counting them inflates affordability, which is the exact direction you do not want your errors to point.

The method that holds up looks for recurring structure:

• Group credits by normalised counterparty, using entity resolution so “ACME CORP PAYROLL”, “ACME CORP”, and “ACME CO” collapse to one employer
• Test each group for regularity in interval (monthly, fortnightly, weekly) and stability in amount
• Separate salary from benefits, pension, rental income and self-employment draws, because credit policy treats them differently
• Distinguish gross-adjacent from net; a statement shows post-tax deposits, so any gross figure you report is a reconstruction and should be labelled as one

Every classified income stream needs lineage back to the specific transactions that produced it. An underwriter, an auditor, or a regulator will ask why the system says a borrower earns a given amount, and “the model decided” is not an answer that survives SR 11-7 model-risk review or an ECOA adverse-action explanation. The transaction list is the answer.

Two failure modes deserve explicit handling. Lookahead: if you use a partial final month that the statement period does not fully cover, you understate or overstate the monthly figure, so clip to complete periods. Drift: employer payment descriptions, statement templates and app export formats change over time, and a parser tuned last quarter quietly degrades. Monitor extraction confidence and reconciliation pass rates as a time series so you see the degradation before a credit committee does.

Benchmarking, or you are flying blind

You cannot claim production accuracy without an eval set that mirrors production. This is the discipline that separates a system you can put in front of a regulator from a script that worked once.

Build the set deliberately. Sample real documents across the banks, formats and channels you actually receive, in the proportions you receive them, and have humans label ground truth: the correct transactions and the correct income figure per stream. Include the ugly cases on purpose. A benchmark made only of clean PDFs measures the part of the problem you already solved.

Then measure at two levels. Field-level extraction accuracy tells you whether the ledger is right. Decision-level accuracy tells you whether the final income figure lands within the tolerance your credit policy allows. They are different questions, and the second is the one that matters for a lending decision.

• Report error on the income figure, not just per-field extraction accuracy
• Track your false-positive budget explicitly: how often the system accepts a wrong figure as confident, since that is the error that reaches an approval
• Calibrate a confidence threshold and route everything below it to manual review, so straight-through processing covers the documents you genuinely parse well and humans catch the tail
• Re-run the benchmark on every model or parser change, and keep the results as part of the audit trail

A statement-derived income figure is an estimate with a confidence band, and the honest version of this system reports both. The number you can trust is the one you can trace to transactions, reconcile against a closing balance, and defend on a held-out set that looks like the customers walking through your door.