Setting security boundaries for sensitive receipt data

Sensitive receipt data leaks downstream whenever a raw OCR payload reaches the policy engine before a validation gate has redacted or quarantined the PII it carries — a PAN fragment, a national tax ID, or an employee home address misread as a merchant descriptor. This page covers the payload-level isolation that stops that leak: the confidence gate, the redaction patterns, the deterministic quarantine states, and the audit-safe fallback that never defaults to allow. It sits inside the Security & Compliance Boundaries component of the broader Core Policy Architecture & Taxonomy Design control plane; where that cluster owns stage-to-stage transitions, this page owns what happens inside a single record. The payloads it inspects arrive from Receipt Ingestion & OCR Data Extraction, and the confidence scores it gates on come from the Tesseract OCR Configuration layer.

Why Standard Approaches Fail

Payload-level boundary failures come from three named modes, each producing a silent PII leak rather than a visible error:

1. Confidence threshold decay. Degraded receipts — crumpled, low-contrast, or multi-currency — force the OCR engine to emit low-confidence tokens. When a pipeline bypasses the threshold to protect throughput, heuristic correction misclassifies a PAN fragment, a regional tax ID, or an employee address as a merchant descriptor, and the unredacted string propagates into analytics and vendor reconciliation. Confidence scoring is owned upstream by Tesseract OCR Configuration; this gate must trust and enforce it, never override it.
2. Unanchored regex evaluation. Pattern matching without word boundaries (\b) or negative lookahead fires false positives on legitimate merchant strings — a street number like 1234 Main St partially matching a PAN pattern. Disabling the rule to silence the noise reopens the leak; the fix is to tighten the assertion, not remove the boundary.
3. Heuristic correction before validation. Raw OCR payloads that reach the policy evaluation engine before structural validation carry unredacted PII straight into routing queues, violating data-minimization mandates. Applying spell-correction or token repair before the boundary check compounds this by inventing plausible-looking sensitive strings out of OCR noise.

The mitigation is a hard quarantine on any payload whose OCR confidence falls below 0.85 or whose unredacted PII patterns match — and never running heuristic correction ahead of boundary validation. Legitimate descriptors that trip a pattern are resolved with negative lookahead or a disambiguation branch, not by relaxing the rule.

Architecture & Algorithm

The enforcement point is a stateless validation function that intercepts each raw payload, applies the gates in fixed precedence, and returns a verdict plus SIEM-compatible audit metadata. Regex patterns are compiled once at module load so parsing cost never recurs per request; @dataclass(slots=True) drops the per-instance __dict__ to keep the memory footprint flat under month-end load. When the engine returns conflicting signals it defaults to the most restrictive boundary and emits a deterministic conflict trace — a SHA-256 of the payload plus the pinned policy_version — so the verdict is replayable against the exact taxonomy it was evaluated under.

import re
import hashlib
import logging
from datetime import datetime, timezone
from typing import Dict, Optional, Tuple
from dataclasses import dataclass, field

# Pre-compile patterns to eliminate per-request regex compilation overhead.
PAN_RE = re.compile(r'\b(?:\d{4}[\s\-]?){3}\d{4}\b')
PII_RE = re.compile(
    r'(?P<ssn>\b\d{3}-\d{2}-\d{4}\b)|'
    r'(?P<tax_id>\b(?:[A-Z]{2}\d{7,10}|IT\d{8})\b)',
    re.IGNORECASE,
)
# Separate pattern to identify address strings (used to reduce false positives on PII_RE).
ADDRESS_RE = re.compile(
    r'\b\d{1,5}\s[A-Za-z\s]{2,}(?:Street|St|Avenue|Ave|Road|Rd)\b', re.IGNORECASE
)

logger = logging.getLogger("receipt_boundary_enforcer")
logger.setLevel(logging.INFO)
_handler = logging.StreamHandler()
_handler.setFormatter(logging.Formatter(
    "%(asctime)s | %(levelname)s | %(name)s | trace_id=%(trace_id)s | %(message)s"
))
logger.addHandler(_handler)


@dataclass(slots=True)
class ReceiptPayload:
    raw_text: str
    ocr_confidence: float
    policy_version: str = "v2.4.1"
    merchant_id: Optional[str] = None
    trace_id: str = field(default_factory=lambda: hashlib.sha256(
        datetime.now(timezone.utc).isoformat().encode()
    ).hexdigest()[:16])


def _deterministic_trace(payload: ReceiptPayload) -> str:
    """Immutable trace ID for audit correlation, pinned to the policy version."""
    return hashlib.sha256(
        f"{payload.policy_version}:{payload.ocr_confidence:.4f}:{payload.raw_text[:128]}".encode()
    ).hexdigest()[:20]


def enforce_security_boundary(
    payload: ReceiptPayload, min_confidence: float = 0.85
) -> Tuple[bool, Dict]:
    """Validate a receipt payload against the PII boundary.

    Returns ``(is_safe, audit_metadata)``. Gates run in fixed precedence and the
    first failure short-circuits, so the most restrictive verdict always wins.
    """
    trace = _deterministic_trace(payload)
    audit: Dict = {
        "trace_id": trace,
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "policy_version": payload.policy_version,
        "status": "pending",
    }

    # Gate 1: confidence threshold — quarantine before any content inspection.
    if payload.ocr_confidence < min_confidence:
        audit.update(status="quarantine", reason="ocr_confidence_below_threshold")
        logger.warning("Payload quarantined: low confidence", extra={"trace_id": trace})
        return False, audit

    # Gate 2: PAN — card numbers are never legitimate in a merchant descriptor.
    if PAN_RE.search(payload.raw_text):
        audit.update(status="quarantine", reason="unredacted_pan_detected", matched_pattern="pan")
        logger.error("Boundary breached: PAN detected", extra={"trace_id": trace})
        return False, audit

    # Gate 3: PII (SSN, tax ID) with address disambiguation to cut false positives.
    pii = PII_RE.search(payload.raw_text)
    if pii:
        if pii.lastgroup == "tax_id" and ADDRESS_RE.search(payload.raw_text):
            # A street address can share a character pattern with a tax ID;
            # route to human review rather than hard-quarantine a valid receipt.
            audit.update(status="review_required", reason="ambiguous_pii_pattern")
            logger.warning("Ambiguous PII pattern; routing for review", extra={"trace_id": trace})
            return False, audit
        audit.update(status="quarantine", reason="unredacted_pii_detected", matched_pattern=pii.lastgroup)
        logger.error("Boundary breached: PII detected", extra={"trace_id": trace})
        return False, audit

    audit.update(status="cleared")
    logger.info("Boundary validation passed", extra={"trace_id": trace})
    return True, audit

A cleared payload is safe to hand to the policy engine; quarantine and review_required are terminal at this boundary and must not advance. The verdict for a given payload and policy_version is fully reproducible — the same bytes always hash to the same trace and reach the same state, regardless of worker or load.

Step-by-Step Integration

1. Pin the confidence source. Confirm ocr_confidence is the value emitted by Tesseract OCR Configuration and not a re-derived estimate. Assert the gate before inspecting content:

   ok, audit = enforce_security_boundary(ReceiptPayload("STORE 12", 0.40))
   assert not ok and audit["reason"] == "ocr_confidence_below_threshold"

2. Redact before, never after. Run enforce_security_boundary on the raw payload before any heuristic correction or normalization. A payload that clears the boundary can then be normalized against the canonical categories in Expense Category Taxonomies; one that fails must not be repaired and retried in-line.

3. Verify the PAN and tax-ID gates independently. Feed known-bad fixtures and assert the exact reason code, so a future regex change cannot silently widen the boundary:

   assert not enforce_security_boundary(ReceiptPayload("Card 4111 1111 1111 1111", 0.99))[0]
   assert enforce_security_boundary(ReceiptPayload("1234 Main St IT12345678", 0.99))[1]["status"] == "review_required"

4. Wire the audit sink first. Route the structured audit dict to your append-only log and confirm trace_id and policy_version appear on every terminal verdict before enabling enforcement — the same immutable-trail requirement the parent Security & Compliance Boundaries component enforces at stage transitions.

5. Fail closed, never open. On timeout, dependency failure, or unhandled exception, the pipeline must route the payload to a dead-letter queue — never default to allow. Capture the failure state, hash the payload, store it in an encrypted bucket with ObjectLock enabled, and emit a SIEM event carrying fallback_action=quarantine and bypass_prevented=true. Reprocess DLQ items only after manual compliance sign-off.

6. Deploy in log-only mode for one cycle. Emit verdicts without blocking, reconcile the quarantine rate against known-sensitive fixtures, then promote to enforcement once the false-positive rate on legitimate merchant strings is acceptable.

Edge Cases & Gotchas

These thresholds should be cross-validated against the access-control and data-minimization controls in NIST SP 800-53 Rev. 5 AC-3, and the pattern-compilation behaviour against the official Python re module documentation.

FAQ

Why quarantine on low confidence instead of just redacting what matches?

Because a low-confidence payload is untrustworthy end to end: the same OCR noise that produced a garbled token can turn a nine-digit tax ID into something the regex never sees. Redaction only removes what you can recognize, so on a degraded scan it leaves unrecognized PII behind. Quarantining below 0.85 and holding for review is the only boundary that does not depend on the very extraction that already failed.

Should the gates run in a different order for performance?

No — precedence is a correctness property, not a tuning knob. The confidence gate runs first because it is the cheapest and rejects the largest share of risky payloads before any content inspection. PAN runs before the broader PII pattern because a card number is never legitimate in a merchant descriptor, so it needs no disambiguation. Reordering to shave microseconds would let a lower-precedence verdict mask a higher-severity one.

How do I stop the tax-ID pattern from flagging real merchant addresses?

Keep the ADDRESS_RE disambiguation branch: when a tax_id match co-occurs with a recognizable street-address pattern, the payload routes to review_required rather than a hard quarantine. Tightening the pattern with negative lookahead is the correct next step if a specific address format keeps tripping it — disabling the rule is not.

What must the fallback do when the validator crashes?

Fail closed. On any timeout, dependency failure, or unhandled exception the payload goes to a dead-letter queue with an immutable, ObjectLock-protected copy and a SIEM event marked bypass_prevented=true. It never defaults to allow, and DLQ items reprocess only after a human compliance sign-off.

Related

• Security & Compliance Boundaries — the parent component whose stage-transition boundaries this payload-level gate feeds
• Classifying OCR extraction errors for manual review — where quarantined and low-confidence payloads are triaged
• Detecting duplicate expenses across overlapping submission windows — a sibling pattern that consumes cleared, redacted records
• Tesseract OCR Configuration — the source of the confidence scores this gate enforces
• Expense Category Taxonomies — canonical categories a cleared payload normalizes to