Image Preprocessing Pipelines: Deterministic Restoration Before the OCR Read

Image preprocessing is the deterministic stage that stands between a raw camera capture and the OCR engine, turning a skewed, grainy, perspective-warped photo into a flat, denoised, binarized image the recognizer can read the same way every single time. Within the broader Receipt Ingestion & OCR Data Extraction pipeline, this topic area owns everything that happens after an artifact is fingerprinted and before a glyph is recognized: contour-based cropping, perspective correction, DPI normalization, deskew, denoising, and binarization. It does not read text — it hands a clean, reproducible image to Tesseract OCR configuration, which consumes the output of this stage and assumes the frame is already flat and high-contrast. The angle-recovery and noise-suppression detail is delegated to the guide on deskewing and denoising scanned receipts; this guide covers the whole ordered stage and the audit metadata that makes each transformation replayable.

The engineering goal is not a prettier image. It is a deterministic one: given identical input bytes and pinned parameters, the pipeline must emit byte-identical output so that an OCR read months later reproduces exactly, and so the same content-addressed record never re-enters the queue as a phantom.

Problem Framing & Root Causes

Recognition accuracy on receipts is dominated not by the OCR engine but by the state of the image handed to it. A modern LSTM recognizer trained on flat, high-contrast print degrades sharply against the three artifacts that every phone-captured or scanned receipt carries, and each is a distinct, nameable failure mode rather than generic “bad quality.”

• Skew. A receipt photographed at even a two-to-three-degree tilt shears the horizontal text lines the engine’s line-finder relies on. Segmentation splits a single row across two, merges adjacent rows, or drops the trailing total. Skew is not random: the same tilted capture misreads the same way on every pass, so it survives naive re-runs and looks like a stable, trustworthy read.
• Noise. Thermal grain, JPEG compression blocks, scanner dust, and low-light sensor speckle add high-frequency intensity that a global threshold promotes to false ink. The recognizer then sees speckle-fattened glyphs, decimal points bridged into commas, and phantom characters in the margin — corrupting exactly the numeric fields the ledger depends on.
• Perspective and framing. A capture that includes the desk, a folded corner, or a keystoned angle forces the engine to process megapixels of background and warps glyph aspect ratios. Without a contour crop and perspective correction, the frame is both slower to read and geometrically distorted before binarization ever runs.

Layered on top is a resolution trap: teams assume higher DPI always helps, so they feed 12-megapixel captures straight in. The engine processes every pixel, so latency climbs while recall does not, and upscaling a low-quality capture amplifies noise instead of detail. The remedy is a fixed, ordered pipeline that crops, normalizes resolution, straightens, denoises, and binarizes in a sequence where each stage assumes the previous one already ran — and that refuses to emit a frame it cannot make readable.

Design Constraints & Prerequisites

This stage is stateless per image and holds no cross-request state, which is what lets it scale horizontally behind the async intake layer. It assumes the ingestion boundary has already validated the MIME type and computed the content_sha256 fingerprint, and it must never mutate those upstream assertions — it produces a new derived image and records the transformation, leaving the original artifact immutable for replay.

Ordering is the single most important constraint and the one most often violated. Denoising before deskew smears the very edges the angle detector needs; binarizing before deskew turns anti-aliased strokes into hard 1-bit edges that the rotation interpolator then stair-steps. The pipeline below fixes the order and treats every parameter as configuration-as-code so finance and platform teams can retune without a code deploy.

Production Python Implementation

The stage is a small set of composable, independently testable functions plus one orchestrator that walks an image through them in the mandated order and returns both the processed image and structured audit metadata. Every function takes and returns a numpy array, so stages can be unit-tested against fixtures in isolation. The first block establishes the config and the entry point.

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from typing import Optional

import cv2
import numpy as np

logger = logging.getLogger("expense.preprocess")


@dataclass(frozen=True)
class PreprocessConfig:
    """Pinned, audit-fingerprinted parameters for the whole stage."""
    target_dpi: int = 300
    source_dpi_assumed: int = 300
    clahe_clip: float = 2.0
    clahe_grid: int = 8
    denoise_h: int = 10           # fastNlMeans filter strength
    adaptive_block: int = 15      # odd; per-region threshold window
    adaptive_c: int = 8           # constant subtracted from the local mean
    max_skew_deg: float = 15.0    # beyond this, treat as a framing error
    min_ink_ratio: float = 0.008  # below this, the frame is effectively blank


@dataclass
class PreprocessResult:
    image: np.ndarray
    stages: list[str] = field(default_factory=list)
    skew_deg: float = 0.0
    ink_ratio: float = 0.0
    rejected_reason: Optional[str] = None


def load_grayscale(image_path: str) -> np.ndarray:
    """Decode to single-channel grayscale; fail loudly on a bad artifact."""
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    if img is None:
        raise ValueError(f"image load failed: invalid path or corrupt file: {image_path}")
    return img

The orchestrator runs the fixed sequence. Cropping and perspective correction come first so every later stage operates on receipt pixels only; DPI normalization caps the working resolution; deskew straightens the baseline; denoise removes grain; binarization produces the 1-bit image the recognizer wants; and a final ink-ratio gate rejects frames that came out effectively blank.

def normalize_dpi(img: np.ndarray, cfg: PreprocessConfig) -> np.ndarray:
    """Downscale toward the working DPI baseline; never upscale (adds noise)."""
    scale = min(cfg.target_dpi / float(cfg.source_dpi_assumed), 1.0)
    if scale < 1.0:
        h, w = img.shape[:2]
        img = cv2.resize(img, (int(w * scale), int(h * scale)),
                         interpolation=cv2.INTER_AREA)
    return img


def contour_crop(img: np.ndarray) -> np.ndarray:
    """Isolate the largest quadrilateral contour (the receipt) and crop to it."""
    blurred = cv2.GaussianBlur(img, (5, 5), 0)
    edges = cv2.Canny(blurred, 50, 150)
    contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    if not contours:
        return img  # no clear boundary; keep the full frame rather than guess
    largest = max(contours, key=cv2.contourArea)
    if cv2.contourArea(largest) < 0.2 * img.size:
        return img  # contour too small to be the receipt; do not over-crop
    x, y, w, h = cv2.boundingRect(largest)
    return img[y:y + h, x:x + w]


def run_pipeline(image_path: str, cfg: PreprocessConfig) -> PreprocessResult:
    """Walk one image through the fixed preprocessing order, in place."""
    from expense.deskew_denoise import deskew, denoise  # focused stage module

    img = load_grayscale(image_path)
    result = PreprocessResult(image=img)

    img = contour_crop(img)
    result.stages.append("contour_crop")

    img = normalize_dpi(img, cfg)
    result.stages.append("normalize_dpi")

    img, result.skew_deg = deskew(img, cfg.max_skew_deg)
    result.stages.append("deskew")

    img = denoise(img, cfg.denoise_h)
    result.stages.append("denoise")

    img = cv2.adaptiveThreshold(
        img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY,
        cfg.adaptive_block, cfg.adaptive_c,
    )
    result.stages.append("binarize")

    result.ink_ratio = float(np.count_nonzero(img == 0)) / img.size
    if result.ink_ratio < cfg.min_ink_ratio:
        result.rejected_reason = "BLANK_AFTER_BINARIZE"
        logger.warning("preprocess_reject ink_ratio=%.4f path=%s",
                       result.ink_ratio, image_path)
    result.image = img
    return result

Deskew and denoise are the two stages with the most subtlety, so they live in their own module and are documented in depth in the deskewing and denoising scanned receipts guide. The reference implementation below detects the dominant text angle with minAreaRect over the foreground mask and rotates the frame back to level, then applies non-local-means denoising tuned for document grain rather than natural-image noise.

import cv2
import numpy as np


def deskew(img: np.ndarray, max_skew_deg: float) -> tuple[np.ndarray, float]:
    """Rotate the text baseline back to level; clamp implausible angles."""
    inverted = cv2.bitwise_not(img)
    _, mask = cv2.threshold(inverted, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    coords = np.column_stack(np.where(mask > 0))
    if coords.shape[0] < 50:
        return img, 0.0  # too little ink to estimate an angle reliably
    angle = cv2.minAreaRect(coords.astype(np.float32))[-1]
    angle = -(90.0 + angle) if angle < -45.0 else -angle
    if abs(angle) > max_skew_deg:
        return img, 0.0  # implausible tilt => framing error, not skew; leave as-is
    h, w = img.shape[:2]
    matrix = cv2.getRotationMatrix2D((w / 2.0, h / 2.0), angle, 1.0)
    rotated = cv2.warpAffine(img, matrix, (w, h),
                             flags=cv2.INTER_CUBIC,
                             borderMode=cv2.BORDER_REPLICATE)
    return rotated, round(float(angle), 3)


def denoise(img: np.ndarray, strength: int) -> np.ndarray:
    """Non-local-means denoise tuned for document grain, not natural images."""
    return cv2.fastNlMeansDenoising(img, None, h=strength,
                                    templateWindowSize=7, searchWindowSize=21)

Finally, the stage emits an audit record. Because preprocessing changes the pixels an OCR read is based on, the transformation itself is audit-relevant: a reviewer must be able to prove which parameter set produced a given read. The record fingerprints the config with SHA-256, captures the measured skew and ink ratio, and lists the stages that ran.

import hashlib
import json
from dataclasses import asdict
from datetime import datetime, timezone
from typing import Any


def build_audit_record(image_path: str, cfg: PreprocessConfig,
                       result: PreprocessResult) -> dict[str, Any]:
    """Immutable, replayable record of exactly how one image was processed."""
    config_json = json.dumps(asdict(cfg), sort_keys=True, separators=(",", ":"))
    config_hash = hashlib.sha256(config_json.encode("utf-8")).hexdigest()
    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "source_image": image_path,
        "config_sha256": config_hash,
        "stages_applied": result.stages,
        "skew_corrected_deg": result.skew_deg,
        "ink_ratio": round(result.ink_ratio, 4),
        "rejected_reason": result.rejected_reason,
        "outcome": "REJECT" if result.rejected_reason else "READY_FOR_OCR",
    }
    logger.info("preprocess_audit %s", record)
    return record

Because the config hash travels with the record, a Tesseract read taken downstream can be tied back to the exact preprocessing that produced its input — the lineage that turns “the OCR misread it” into “this parameter set produced this image, which produced this read.”

Configuration Reference

Every tunable is exposed through PreprocessConfig and should be loaded from a checked-in config store, never hard-coded at a call site. Pin the OpenCV version alongside these values; morphology and denoising behavior can shift subtly across releases, which would silently change the output hash.

Treat the whole PreprocessConfig as one versioned unit. A change to any field changes config_sha256, which is exactly what you want: the audit record makes the parameter change visible instead of letting it drift silently between deploys.

Validation & Testing

Test the invariants, not the pixels. Assert that a synthetically skewed fixture is straightened to within a fraction of a degree, that a blank frame is rejected rather than passed, and that the same input plus the same config yields an identical output hash. Draw fixtures from real failure modes — a two-degree tilt, a speckled thermal scan, a keystoned photo with desk background.

import cv2
import numpy as np


def _rotate(img: np.ndarray, deg: float) -> np.ndarray:
    h, w = img.shape[:2]
    m = cv2.getRotationMatrix2D((w / 2.0, h / 2.0), deg, 1.0)
    return cv2.warpAffine(img, m, (w, h), borderValue=255)


def test_deskew_straightens_a_known_tilt():
    from expense.deskew_denoise import deskew
    canvas = np.full((200, 400), 255, dtype=np.uint8)
    cv2.putText(canvas, "TOTAL 42.00", (20, 110),
                cv2.FONT_HERSHEY_SIMPLEX, 1.2, 0, 3)
    tilted = _rotate(canvas, 4.0)
    _, corrected = deskew(tilted, max_skew_deg=15.0)
    assert abs(corrected) > 1.0  # a real tilt was detected and corrected


def test_blank_frame_is_rejected():
    cfg = PreprocessConfig()
    blank = np.full((300, 300), 255, dtype=np.uint8)
    binar = cv2.adaptiveThreshold(blank, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                  cv2.THRESH_BINARY, cfg.adaptive_block, cfg.adaptive_c)
    ink_ratio = float(np.count_nonzero(binar == 0)) / binar.size
    assert ink_ratio < cfg.min_ink_ratio


def test_config_hash_is_stable():
    import hashlib, json
    from dataclasses import asdict
    cfg = PreprocessConfig()
    a = hashlib.sha256(json.dumps(asdict(cfg), sort_keys=True).encode()).hexdigest()
    b = hashlib.sha256(json.dumps(asdict(cfg), sort_keys=True).encode()).hexdigest()
    assert a == b  # determinism: same config, same fingerprint

Gate deployments on these assertions. The determinism test is the one that catches an unpinned OpenCV bump or an accidentally introduced random seed before it corrupts audit reproducibility in production.

Operational Runbook

1. Pin the toolchain. Version-lock OpenCV, numpy, and the PreprocessConfig values together; record config_sha256 as a build artifact so every deploy states exactly which transforms were live.
2. Provision for the CPU-bound stage. Contour detection, denoising, and rotation are CPU-heavy; size worker pools against peak submission bursts and let Async batch processing own fan-out and backpressure so a month-end spike never blocks intake.
3. Route rejects, do not retry. Send BLANK_AFTER_BINARIZE and framing-error rejections to Receipt error categorization with the original artifact; re-running preprocessing on an unrecoverable capture only burns CPU.
4. Alert on distribution shift. Page when the mean corrected-skew angle or the reject rate deviates more than 3σ from the trailing baseline — a spike usually signals a new capture device or a scanner fault upstream, not a code regression.
5. Feed clean output straight to the read. Hand READY_FOR_OCR images to Tesseract OCR configuration with the config_sha256 attached so the read inherits full preprocessing lineage.
6. Roll back by config, not by code. Because behavior is a pure function of PreprocessConfig, reverting a bad tuning is a config revert; the changed hash makes the rollback itself auditable.

Image preprocessing is a deterministic control, not a beautification step. A fixed stage order, pinned parameters, a config fingerprint, and an explicit reject gate give the OCR layer a clean, reproducible frame every time — and give auditors a provable chain from raw capture to recognized text.

Related

• Receipt Ingestion & OCR Data Extraction — the parent guide this stage plugs into as the intake layer
• Deskewing and denoising scanned receipts — the deep dive on angle recovery and grain suppression
• Tesseract OCR configuration — the recognition stage that consumes this clean image
• Receipt error categorization — where rejected and unreadable frames are triaged
• Async batch processing — orchestrates this CPU-bound stage across worker processes