Expense Audit Guide Python · Pipelines · Compliance

Deskewing and denoising scanned receipts before OCR

A receipt scanned even a few degrees off-axis and speckled with thermal grain will read wrong on every pass, because the OCR line-finder shears skewed rows apart and a global threshold promotes noise to false ink — and the order you apply the fixes in decides whether they help or hurt. This guide is the angle-recovery and grain-suppression detail delegated by the image preprocessing pipelines stage, which itself sits inside the broader Receipt Ingestion & OCR Data Extraction intake layer. The parent stage owns the full crop-normalize-deskew-denoise-binarize sequence; here we own two of those steps in depth: measuring and correcting skew with OpenCV, suppressing noise without erasing thin strokes, and — critically — the fixed ordering that keeps each step from destroying the signal the next one needs.

The failure this page prevents is subtle: a skewed, grainy scan does not error out. It produces a confident, plausible-looking read with the decimal in the wrong place or the date row merged into the total — a deterministic corruption that survives naive re-runs.

Deskew-then-denoise ordering feeding a clean image to OCR A grayscale receipt scan passes left to right through five stages in a fixed order. Stage one builds a foreground mask by inverting and Otsu-thresholding. Stage two estimates the skew angle using minAreaRect over the ink pixels with a Hough-line cross-check. Stage three rotates the frame to level with a cubic warpAffine. Stage four denoises with fastNlMeans tuned for document grain. Stage five applies a morphological open, an erosion followed by a dilation, to clear residual speckle. The output emphasises that deskew must run before denoise and denoise before binarize, because that order preserves the edges each stage depends on. Straighten first, then clean — the order preserves the edges Denoise before deskew smears the edges the angle detector needs; binarize before deskew stair-steps strokes. 1 · Foreground mask invert · Otsu 2 · Angle estimate minAreaRect · Hough 3 · Rotate to level warpAffine · cubic 4 · Denoise fastNlMeans 5 · Morph open erode then dilate Level, grain-free image → adaptive binarization → Tesseract read deskew before denoise · denoise before binarize · every rotation angle logged for audit

Why Standard Approaches Fail

Most teams reach for a single OpenCV call and assume the problem is solved. Three concrete patterns explain why that read still comes back corrupted.

  • Denoising before deskew smears the angle signal. A skew detector estimates the tilt from the geometry of the ink — the long axis of the text block or the slope of its baselines. Run fastNlMeans or a Gaussian blur first and you soften exactly those edges, so minAreaRect and Hough both return a wobblier angle. On a real AP scan the correction lands a degree short, the total row still shears, and the denoise “fix” has quietly made the deskew worse.
  • Binarizing before deskew stair-steps the strokes. Threshold first and the anti-aliased grey edges of each glyph become hard 1-bit boundaries. Rotating a 1-bit image then forces the interpolator to choose black or white per pixel along a diagonal, producing jagged, stair-stepped strokes that the recognizer misreads — a 5 fringes into an 8, a decimal point smears into the digit beside it.
  • One denoiser for every kind of noise. A median blur that clears salt-and-pepper thermal speckle will also dissolve the thin strokes of a small-font line item, and a fastNlMeans strength tuned for a clean office scan leaves a low-light phone capture still grainy. Treating “noise” as one undifferentiated thing over-cleans the faint receipts and under-cleans the loud ones, and both outcomes land in the reject queue of the downstream Tesseract OCR configuration stage.

The fix is not a better single call. It is the correct sequence — mask, estimate, rotate, denoise, open — with each step tuned for documents rather than photographs, and with the measured angle recorded so a bad correction is visible in the audit trail rather than hidden inside a plausible read.

Architecture & Algorithm

Skew estimation is most robust when two independent methods agree. minAreaRect fits the tightest rotated rectangle around every foreground pixel and reads its angle; a Hough-line vote finds the dominant near-horizontal text lines and reads their median slope. Using minAreaRect as the primary estimate and Hough as a sanity cross-check catches the case where a large logo or a table rule biases the bounding rectangle away from the true text angle.

from __future__ import annotations

import logging

import cv2
import numpy as np

logger = logging.getLogger("expense.deskew")


def _foreground_mask(gray: np.ndarray) -> np.ndarray:
    """Invert so ink is white, then Otsu-threshold to a clean binary mask."""
    inverted = cv2.bitwise_not(gray)
    _, mask = cv2.threshold(inverted, 0, 255,
                            cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    return mask


def estimate_skew_minarearect(mask: np.ndarray) -> float:
    """Angle of the tightest rotated rectangle around all ink pixels."""
    coords = np.column_stack(np.where(mask > 0))
    if coords.shape[0] < 50:
        return 0.0
    angle = cv2.minAreaRect(coords.astype(np.float32))[-1]
    return -(90.0 + angle) if angle < -45.0 else -angle


def estimate_skew_hough(mask: np.ndarray) -> float | None:
    """Median slope of dominant near-horizontal text lines, or None."""
    lines = cv2.HoughLinesP(mask, 1, np.pi / 180, threshold=100,
                            minLineLength=mask.shape[1] // 3, maxLineGap=20)
    if lines is None:
        return None
    angles: list[float] = []
    for x1, y1, x2, y2 in lines[:, 0]:
        theta = np.degrees(np.arctan2(y2 - y1, x2 - x1))
        if abs(theta) < 30.0:            # keep near-horizontal lines only
            angles.append(theta)
    return float(np.median(angles)) if angles else None

The correction routine combines the two estimates, clamps implausible angles as framing errors rather than skew, and rotates with a cubic interpolation and replicated border so the corners do not fill with black. It returns the corrected image and the applied angle so the caller can log it.

def deskew(gray: np.ndarray, max_skew_deg: float = 15.0) -> tuple[np.ndarray, float]:
    """Deskew a grayscale scan; return the image and the angle applied."""
    mask = _foreground_mask(gray)
    primary = estimate_skew_minarearect(mask)
    cross = estimate_skew_hough(mask)

    # If Hough disagrees sharply, the bounding rect was biased (logo/rule);
    # trust the text-line vote instead.
    angle = primary
    if cross is not None and abs(cross - primary) > 2.0:
        angle = cross

    if abs(angle) > max_skew_deg:
        logger.warning("skew_out_of_range angle=%.2f -> treated as framing error", angle)
        return gray, 0.0

    h, w = gray.shape[:2]
    matrix = cv2.getRotationMatrix2D((w / 2.0, h / 2.0), angle, 1.0)
    rotated = cv2.warpAffine(gray, matrix, (w, h),
                             flags=cv2.INTER_CUBIC,
                             borderMode=cv2.BORDER_REPLICATE)
    return rotated, round(float(angle), 3)

Denoising runs only after the frame is level. fastNlMeans handles Gaussian sensor and compression grain while preserving stroke edges better than a blur; a light morphological open — an erosion followed by a dilation with a small kernel — then clears any residual isolated speckle without thickening glyphs. The combined restoration keeps the two operations in the one order that works.

def restore_for_ocr(gray: np.ndarray, denoise_h: int = 10) -> tuple[np.ndarray, float]:
    """Deskew, then denoise, then open — the only order that preserves strokes."""
    leveled, applied_angle = deskew(gray)
    denoised = cv2.fastNlMeansDenoising(leveled, None, h=denoise_h,
                                        templateWindowSize=7, searchWindowSize=21)
    kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))
    opened = cv2.morphologyEx(denoised, cv2.MORPH_OPEN, kernel)
    logger.info("restore_for_ocr applied_angle=%.3f denoise_h=%d", applied_angle, denoise_h)
    return opened, applied_angle

Because restore_for_ocr returns the applied angle, the caller records it in the same audit envelope the parent stage maintains, so a downstream misread can always be traced to the exact rotation and denoise strength that produced its input.

Step-by-Step Integration

  1. Take grayscale input from the crop stage. Feed restore_for_ocr the already-cropped, DPI-normalized grayscale image from the image preprocessing pipelines orchestrator — never a raw color photo, which triples memory and adds no monochrome signal.

  2. Deskew before you denoise, always. Call deskew first so the angle estimate reads sharp edges; only then denoise. Never reorder these for “efficiency.”

  3. Keep binarization last. Do not threshold before rotating; the adaptive binarization belongs in the parent stage, after restoration returns.

  4. Log the applied angle. Persist applied_angle into the audit record so an unusually large correction is visible, and alert if the mean angle drifts — it usually signals a new capture device.

  5. Verify against a known tilt before wiring it in. Confirm a synthetically skewed fixture is straightened and that denoising ran without erasing strokes:

    import cv2
import numpy as np

canvas = np.full((220, 460), 255, dtype=np.uint8)
cv2.putText(canvas, "TOTAL 42.00", (24, 120),
            cv2.FONT_HERSHEY_SIMPLEX, 1.3, 0, 3)
m = cv2.getRotationMatrix2D((230, 110), 5.0, 1.0)      # induce a 5-degree tilt
tilted = cv2.warpAffine(canvas, m, (460, 220), borderValue=255)

restored, angle = restore_for_ocr(tilted)
assert abs(angle) > 1.5                                 # a real tilt was corrected
assert np.count_nonzero(restored == 0) > 100            # strokes survived denoise

  6. Route irrecoverable frames onward, do not loop. If a frame is still unreadable after restoration, hand it to Receipt error categorization rather than re-running the pipeline on a capture that cannot improve.

Edge Cases & Gotchas

Edge condition What breaks Mitigation
Large logo or table rule minAreaRect angle biased away from the text baseline Cross-check with the Hough text-line vote; prefer it when the two disagree by more than 2°
Near-blank frame Too few ink pixels for a stable angle estimate Guard on foreground pixel count (< 50) and return 0° rather than a spurious rotation
Upside-down or 90° capture Deskew corrects small tilt, not orientation Clamp beyond max_skew_deg as a framing error and route to orientation detection, not deskew
Over-strong denoise High fastNlMeans h erases thin strokes on faint receipts Keep h at 10 as a default; lower it for faded substrates instead of raising it
Median blur on small fonts Salt-and-pepper filter dissolves 6–8px glyphs Use fastNlMeans plus a (2,2) open, not a wide median kernel
Black corners after rotation Zero-fill borders threshold to false ink Rotate with BORDER_REPLICATE so corners inherit edge pixels
Denoise applied before deskew Softened edges yield a wobbly, short angle estimate Enforce the fixed order in restore_for_ocr; never expose a reordered path

FAQ

Should I deskew or denoise first?

Deskew first, then denoise. The skew estimator reads the geometry of the ink edges, and denoising softens exactly those edges, so denoising first gives you a wobblier, usually short angle estimate. Straightening on sharp edges and then cleaning the level frame is the order that preserves accuracy, which is why restore_for_ocr hard-codes it.

minAreaRect or Hough transform for the skew angle?

Use minAreaRect as the primary estimate and a Hough-line vote as a cross-check. minAreaRect is fast and robust when the ink is mostly text, but a large logo or a heavy table rule can bias its bounding rectangle. When the Hough median of the near-horizontal text lines disagrees with it by more than about two degrees, trust the text-line vote, because it measures the baselines you actually care about.

Why not just binarize and be done with it?

Binarizing before deskew turns anti-aliased strokes into hard 1-bit edges, and rotating that 1-bit image stair-steps every diagonal, corrupting glyphs. Binarization has to come after restoration. Keep thresholding in the parent stage, run deskew and denoise on the grayscale frame first, and let the adaptive threshold see clean, level strokes.

What denoise strength should I use for receipts?

Start with a fastNlMeans h of 10 and a (2,2) morphological open. That clears typical thermal grain and compression blocks while preserving thin strokes. For faded or low-contrast receipts, lower h rather than raising it — a stronger filter erases the faint ink you are trying to recover — and let the faded-text restoration path handle contrast recovery separately.

Does the deskew angle need to be audited?

Yes. Preprocessing changes the pixels the OCR read is based on, so the applied rotation is audit-relevant. Log applied_angle in the same record the intake stage keeps, and alert when the mean correction drifts from its baseline. An unexplained shift usually means a new scanner or camera upstream, and having the angle in the ledger lets you prove which transform produced any given read.