Expense Audit Guide Python · Pipelines · Compliance

Improving table extraction accuracy in pdfplumber

pdfplumber.extract_tables() returns clean rows on ruled invoices and collapses to fragmented noise on the borderless, whitespace-aligned receipts that dominate real expense corpora — the fix is to stop trusting the default table settings and tune the strategy, snap, and join tolerances per document class. This guide extends the parent pdfplumber Line-Item Parsing guide inside the broader Receipt Ingestion & OCR Data Extraction framework, and it targets the single most common source of silent under-extraction: a table_settings dictionary copied from a tutorial and never matched to the vendor’s actual page geometry.

The objective is deterministic column recovery — the same folio parsed on two workers must yield the same grid — so that every amount reaches Automated Policy Validation & Anomaly Flagging with a stable row structure behind it.

Adaptive pdfplumber table-strategy selection with a column-grid stability gate A vector PDF page carrying characters, lines, and rectangles enters a decision that inspects whether ruling lines are present. If ruled edges exist, the lines strategy snaps columns to the vector rules; if the page is borderless the text or explicit strategy derives columns from word alignment. Both strategies feed a stage that tunes snap, join, and edge-minimum-length tolerances. The tuned grid enters a stability gate asking whether the column count is consistent across rows. A stable grid passes to typed line-item rows carrying Decimal amounts and provenance; an unstable grid is flagged as column drift and routed to re-tuning or the exception queue. Choose the table strategy from page geometry, then gate on grid stability Vector page chars · lines · rects ruling lines? edges vs whitespace ruled borderless lines strategy snap columns to vector rules text / explicit strategy derive columns from words tune snap & join tolerances snap · join · edge_min_length grid stable? columns aligned consistent drift Typed line-item rows amount · Decimal · provenance Column drift flagged re-tune or exception queue

Why Standard Approaches Fail

The default table_settings assume a fully ruled table, and every real expense corpus violates that assumption in a way that produces a silent loss rather than a crash. Three named failure modes recur across accounts-payable traffic.

  • Strategy mismatch on borderless layouts. The default vertical_strategy="lines" needs vector ruling edges to snap to. A restaurant check or a modern SaaS invoice that aligns columns with whitespace has none, so extract_tables() returns None or a single smeared column. Downstream code reads that as a clean zero-line-item receipt and lets it pass policy evaluation unexamined.
  • Snap tolerance too tight, so columns drift. Anti-aliased or slightly skewed scans render “vertical” rules a fraction of a point off true. With snap_x_tolerance at its default, two segments of the same rule are treated as two columns on one row and one column on the next, so the same document yields four columns near the top and five near the bottom — the amount column lands under the description column halfway down the page.
  • Join tolerance collapsing adjacent columns. Dense thermal folios pack the quantity, unit price, and line total close together. An over-wide join_x_tolerance merges those cells into one, and the amount parser then sees 2 4.50 9.00 as a single token. The row survives, the number is wrong, and nothing flags it because a value was extracted.

Each failure is deterministic: it reproduces on every run, so a naive retry never clears it. The remedy is to select the strategy from the page’s own geometry and pin the tolerance set per document class.

Architecture & Algorithm

The extractor below inspects each page for ruling edges, picks a lines or a text strategy accordingly, applies a version-controlled tolerance profile, and then measures column-count stability across the recovered rows so that a drifting grid is flagged instead of trusted. Monetary values are parsed with decimal.Decimal to prevent float drift, and every document emits a structured report that lets an auditor reconstruct which strategy and tolerance set produced a given grid.

from __future__ import annotations

import json
import logging
import re
from collections import Counter
from dataclasses import dataclass, asdict
from decimal import Decimal, InvalidOperation
from typing import Any, Dict, List, Optional, Tuple

import pdfplumber

logger = logging.getLogger("expense_audit.table_extraction")
if not logger.handlers:
    logger.setLevel(logging.INFO)
    _handler = logging.StreamHandler()
    _handler.setFormatter(
        logging.Formatter('{"ts":"%(asctime)s","lvl":"%(levelname)s","msg":"%(message)s"}')
    )
    logger.addHandler(_handler)

# Two pinned table_settings profiles. Ruled vendors snap columns to vector
# lines; borderless vendors have no rulings, so columns are inferred from the
# horizontal alignment of words. Keep both in version control, not in code.
RULED_SETTINGS: Dict[str, Any] = {
    "vertical_strategy": "lines",
    "horizontal_strategy": "lines",
    "snap_x_tolerance": 3,     # merge near-collinear vertical rules within 3pt
    "snap_y_tolerance": 3,
    "join_x_tolerance": 3,     # keep small: a wide join collapses columns
    "join_y_tolerance": 3,
    "edge_min_length": 12,     # ignore stray tick marks shorter than a real rule
    "intersection_tolerance": 3,
}
BORDERLESS_SETTINGS: Dict[str, Any] = {
    "vertical_strategy": "text",
    "horizontal_strategy": "text",
    "text_x_tolerance": 2,
    "text_y_tolerance": 3,
    "snap_x_tolerance": 6,     # looser: word-derived columns are noisier
    "min_words_vertical": 2,   # a column must contain >=2 aligned words
    "min_words_horizontal": 1,
}

_AMOUNT_RE = re.compile(r"(?:[$€£¥]|USD|EUR|GBP)?\s*(\d[\d,]*\.\d{2})")


@dataclass(frozen=True)
class TableRow:
    """Immutable extracted row with the strategy that produced it."""

    cells: Tuple[str, ...]
    amount: Optional[Decimal]
    page_index: int
    row_index: int
    strategy: str
    column_count: int


@dataclass
class ExtractionReport:
    """Per-document audit envelope for the table-extraction decision."""

    correlation_id: str
    strategy: str
    row_count: int
    modal_columns: int
    drift_rows: int
    stable: bool


def _has_ruling_lines(page, min_edges: int = 4) -> bool:
    """True when the page carries enough vector edges to trust `lines`."""
    verticals = [e for e in page.edges if e.get("orientation") == "v"]
    horizontals = [e for e in page.edges if e.get("orientation") == "h"]
    return len(verticals) >= min_edges and len(horizontals) >= min_edges


def _parse_amount(cells: Tuple[str, ...]) -> Optional[Decimal]:
    """Return the first Decimal-parseable currency value in a row, or None."""
    for cell in cells:
        match = _AMOUNT_RE.search(cell)
        if match:
            try:
                return Decimal(match.group(1).replace(",", ""))
            except InvalidOperation:
                return None
    return None


def extract_page_rows(page, page_idx: int) -> Tuple[List[TableRow], str]:
    """Pick a strategy from geometry and extract typed rows from one page."""
    ruled = _has_ruling_lines(page)
    settings = RULED_SETTINGS if ruled else BORDERLESS_SETTINGS
    strategy = "lines" if ruled else "text"

    rows: List[TableRow] = []
    row_idx = 0
    for table in page.extract_tables(settings) or []:
        for raw in table:
            cells = tuple((c or "").strip() for c in raw)
            populated = [c for c in cells if c]
            if not populated:
                continue
            rows.append(
                TableRow(
                    cells=cells,
                    amount=_parse_amount(cells),
                    page_index=page_idx,
                    row_index=row_idx,
                    strategy=strategy,
                    column_count=len(populated),
                )
            )
            row_idx += 1
    return rows, strategy


def assess_stability(rows: List[TableRow], drift_ratio: float = 0.15) -> Tuple[int, int, bool]:
    """Measure column-count consistency; drift above the ratio is unstable."""
    counts = [r.column_count for r in rows if r.column_count]
    if not counts:
        return 0, 0, False
    modal = Counter(counts).most_common(1)[0][0]
    drift = sum(1 for c in counts if c != modal)
    stable = drift <= len(counts) * drift_ratio
    return modal, drift, stable


def extract_document(pdf_path: str, correlation_id: str) -> Tuple[List[TableRow], ExtractionReport]:
    """Extract every page, then report grid stability for the whole document."""
    all_rows: List[TableRow] = []
    strategy = "none"
    with pdfplumber.open(pdf_path) as pdf:
        for page_idx, page in enumerate(pdf.pages):
            page_rows, strategy = extract_page_rows(page, page_idx)
            all_rows.extend(page_rows)

    modal, drift, stable = assess_stability(all_rows)
    report = ExtractionReport(
        correlation_id=correlation_id,
        strategy=strategy,
        row_count=len(all_rows),
        modal_columns=modal,
        drift_rows=drift,
        stable=stable,
    )
    logger.info(json.dumps(asdict(report)))
    return all_rows, report

The stability check is the audit-critical addition: a page can return rows and still be wrong if the column count wanders, so assess_stability compares each row’s populated-cell count against the modal grid width and marks the document unstable when too many rows disagree. A frozen TableRow means no downstream stage can rewrite a cell after extraction, and the ExtractionReport records exactly which strategy and tolerance profile produced the grid.

Step-by-Step Integration

  1. Classify the vendor before you tune. Group incoming documents by issuer and inspect a sample with page.edges and page.debug_tablefinder(). Ruled vendors go to RULED_SETTINGS; whitespace-aligned vendors go to BORDERLESS_SETTINGS. Never share one tolerance set across both.

  2. Pin the toolchain and the profiles together. A pdfplumber upgrade can shift default word grouping, which moves borderless columns. Pin the library and treat the two settings dictionaries as versioned configuration:

    pip install "pdfplumber==0.11.4" "pdfminer.six==20231228"

  3. Prefer flipping the strategy over loosening tolerances. When a ruled profile under-extracts, switch that vendor to text before widening join_x_tolerance; a wide join silently collapses columns for every document of that class at once.

  4. Assert grid stability, not just non-empty output. A row count above zero is not success. Verify the recovered grid is consistent before wiring the rows into the ledger:

    rows, report = extract_document("fixtures/borderless_restaurant_check.pdf", "EXP-9001")
assert rows, "no rows recovered — strategy mismatch, check page.edges"
assert report.stable, f"column drift: {report.drift_rows} of {report.row_count} rows off-grid"
assert all(r.amount is None or r.amount > 0 for r in rows)

  5. Cross-check totals against a second reader. Because table geometry can succeed while a single amount is mis-joined, reconcile the extracted line items against the receipt total and against Parsing multi-page PDF receipts and folios for documents that span pages, then hand identical line-item sets to duplicate receipt detection so a re-submitted folio is caught on its rows, not just its total.

  6. Route unstable grids to review. Send documents where report.stable is false to the queue owned by Receipt Error Categorization with the strategy and drift metrics attached, rather than posting an off-grid amount.

Edge Cases & Gotchas

Edge condition What breaks Mitigation
Borderless vendor sent to lines strategy extract_tables() returns None; zero rows read as a clean receipt Gate on page.edges count and route to text when rulings are absent
Faint or dashed ruling lines Edges fall below edge_min_length and are dropped, splitting columns Lower edge_min_length for that vendor, or add explicit_vertical_lines
Skewed scan, near-collinear rules snap_x_tolerance too tight splits one rule into two columns Raise snap_x_tolerance a few points; deskew upstream before parsing
Adjacent price columns merged Wide join_x_tolerance fuses quantity and total into one cell Keep join_x_tolerance small; validate row column count against the modal grid
Multi-line wrapped description Wrapped text emits an extra row with no amount Treat amount-less rows as continuations, not new line items
Nested sub-tables (tax breakdown) extract_tables() returns two overlapping grids Restrict extraction to a bounding box with page.crop() per region
Currency symbol in its own column Amount regex misses the value split from its symbol Join the row cells before running _AMOUNT_RE, not each cell alone

Because silent grid changes would undermine SOX Section 404 control expectations, the strategy, modal column count, and drift count are logged on every document so the extraction decision is reproducible from the pinned configuration.

FAQ

When should I use the lines strategy versus the text strategy?

Use lines when the vendor draws real vector ruling edges you can see in page.edges — snapping columns to those rules is the most accurate option available. Use text when the layout is borderless and columns are held together by whitespace alignment only, because there are no rules for lines to find. Detect which case you are in programmatically by counting vertical and horizontal edges rather than guessing per vendor.

Why does raising join tolerance sometimes lose columns?

join_x_tolerance controls how far apart two cell edges can be and still be treated as the same column boundary. Set it too wide and pdfplumber merges genuinely separate columns — quantity, unit price, and line total collapse into one cell — so a value is still extracted but it is wrong. Keep the join tolerance small and fix under-extraction by switching strategy or adding explicit lines instead.

How do I extract a borderless table with no ruling lines at all?

Switch that vendor to vertical_strategy="text" so columns are inferred from the horizontal positions of words, and set min_words_vertical so a column must contain at least a couple of aligned words before it is accepted. If the alignment is reliable but the automatic inference wobbles, supply explicit_vertical_lines with the known x-coordinates of each column to pin the grid exactly.

My columns drift halfway down the page — what causes that?

Column drift almost always comes from a snap tolerance that is too tight for a slightly skewed scan: two segments of the same vertical rule render a fraction of a point apart and get treated as different columns on different rows. Raise snap_x_tolerance a few points, deskew the image upstream, and use the column-count stability check to catch any residual drift before the rows reach the ledger.

Should table settings be the same across every vendor?

No. A single tolerance set that works for a ruled hotel folio will under-extract a borderless SaaS invoice and over-merge a dense thermal receipt. Classify documents by issuer, pin a settings profile per class, and store those profiles in version control so a change is a reviewed, logged event rather than an untracked edit that silently re-parses your history.