Detecting stuck approvals with SLA timers

A reimbursement wedged in PENDING_APPROVAL rarely fails loudly — it simply sits, because the one approver assigned to it is on leave and no clock is watching, so the report ages in silence until the employee escalates by hand. This guide is the focused detection pattern delegated by the Approval SLA Monitoring topic area, which sits inside the broader Reimbursement Routing & Approval Sync framework. The parent guide defines the budgets and metrics; here we build the narrow thing that runs on a timer: a deterministic sweep that finds every report still pending past its SLA deadline, reconstructs exactly how long it has been stuck from the append-only audit ledger, and fires escalation once — with a trail that proves why.

Why Standard Approaches Fail

Teams reach for the obvious query first — “show me reports older than N days” — and it fails on stuck approvals for three concrete reasons.

• Age-since-submission is the wrong clock. A WHERE submitted_at < now() - interval '3 days' filter flags a report that has been bouncing productively through three reviewers as urgently as one genuinely frozen in a single approver’s queue. The report that matters is the one that has not changed state in too long, not the one that is simply old. Measuring from submitted_at buries the real stalls under a pile of healthy in-flight reports.
• A mutable updated_at column lies after replays. Systems that stamp a single updated_at on the report row lose the truth the moment a background job, a comment, or an ERP sync touches the row without changing the approval state — the timestamp moves and the stuck report looks fresh again. The only trustworthy source of “when did it enter this state” is the immutable transition event in the audit ledger, not a field that anything can overwrite.
• Naive re-runs double-fire or go silent. A sweep with no idempotency either pages every interval for the same stuck report — training the approver to mute the channel — or, when someone adds a crude “already alerted” flag on the row, goes permanently silent after a rejection-and-resubmit puts the report back into PENDING_APPROVAL with the stale flag still set. Detection has to key on the specific state entry, so a fresh stall after re-entry pages again but a single stall pages once.

The fix is a sweep that derives time-in-state from the ledger, compares it to a deadline computed from the tier budget, and deduplicates on the exact state entry rather than the report id.

Architecture & Algorithm

The scanner is a stateless function over three inputs: the set of currently open reports, a reader that returns each report’s transition history, and an injected clock. For every open report it finds the latest transition into PENDING_APPROVAL, computes the deadline as that instant plus the tier budget, and yields a StuckApproval only when the clock is past the deadline and the report is still pending. Injecting the clock is what makes the sweep replayable — the same ledger and the same now() always produce the same stuck set.

from __future__ import annotations

import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Callable, Iterable, Iterator, Mapping, Sequence

logger = logging.getLogger("expense.sla.stuck")

PENDING = "PENDING_APPROVAL"


@dataclass(frozen=True)
class LedgerEvent:
    """One immutable transition row from the approval audit ledger."""
    report_id: str
    to_state: str
    approval_tier: str
    event_ts: datetime  # tz-aware UTC


@dataclass(frozen=True)
class StuckApproval:
    report_id: str
    approval_tier: str
    entered_pending_at: datetime
    deadline: datetime
    overdue_by: timedelta
    dedup_key: str


def _latest_pending_entry(events: Sequence[LedgerEvent]) -> LedgerEvent | None:
    """Return the most recent transition INTO PENDING_APPROVAL, or None.

    A rejection loop can enter PENDING more than once; only the latest,
    contiguous entry defines the current dwell, so earlier waits never count.
    """
    pending = [e for e in events if e.to_state == PENDING]
    if not pending:
        return None
    latest = max(pending, key=lambda e: e.event_ts)
    # If a later transition moved the report OUT of PENDING, it is not stuck.
    later = [e for e in events if e.event_ts > latest.event_ts]
    if any(e.to_state != PENDING for e in later):
        return None
    return latest


def scan_for_stuck(
    open_report_ids: Iterable[str],
    history_of: Callable[[str], Sequence[LedgerEvent]],
    tier_budgets: Mapping[str, timedelta],
    now: Callable[[], datetime],
) -> Iterator[StuckApproval]:
    """Yield one StuckApproval per report past its PENDING_APPROVAL deadline."""
    current = now().astimezone(timezone.utc)
    for report_id in open_report_ids:
        entry = _latest_pending_entry(history_of(report_id))
        if entry is None:
            continue
        budget = tier_budgets.get(entry.approval_tier)
        if budget is None:
            logger.warning("no_budget_for_tier", extra={"report_id": report_id,
                                                         "tier": entry.approval_tier})
            continue  # diverts to review; never assume a default deadline
        deadline = entry.event_ts + budget
        if current <= deadline:
            continue  # still within SLA
        # Dedup on the exact state entry, so one stall pages once but a
        # re-entry after rejection is a new key and pages again.
        dedup_key = f"{report_id}:{entry.event_ts.isoformat()}"
        yield StuckApproval(
            report_id=report_id,
            approval_tier=entry.approval_tier,
            entered_pending_at=entry.event_ts,
            deadline=deadline,
            overdue_by=current - deadline,
            dedup_key=dedup_key,
        )

The escalation step is deliberately thin and side-effect-honest: it records the dedup key before signalling, so a crash mid-sweep cannot escalate the same stall twice on the next run, and it appends an audit row carrying the inputs that justified the alert.

from __future__ import annotations

import hashlib
import json
import logging
from datetime import datetime, timezone
from typing import Callable, Protocol, Set

logger = logging.getLogger("expense.sla.escalate")


class Escalator(Protocol):
    def escalate(self, *, report_id: str, tier: str, overdue_hours: float) -> None: ...


def escalate_stuck(stuck: StuckApproval, escalator: Escalator,
                   already_paged: Set[str], now: Callable[[], datetime]) -> dict | None:
    """Escalate a stuck approval exactly once and return its audit row.

    `already_paged` is the persisted set of dedup keys. Recording the key
    BEFORE escalating makes a re-run after a crash a no-op for this entry.
    """
    if stuck.dedup_key in already_paged:
        return None
    already_paged.add(stuck.dedup_key)

    overdue_hours = round(stuck.overdue_by.total_seconds() / 3600.0, 2)
    row = {
        "event": "STUCK_APPROVAL_ESCALATED",
        "report_id": stuck.report_id,
        "approval_tier": stuck.approval_tier,
        "entered_pending_at": stuck.entered_pending_at.isoformat(),
        "deadline": stuck.deadline.isoformat(),
        "overdue_hours": overdue_hours,
        "detected_at": now().astimezone(timezone.utc).isoformat(),
        "dedup_key": stuck.dedup_key,
    }
    row["audit_hash"] = hashlib.sha256(
        json.dumps(row, sort_keys=True).encode("utf-8")
    ).hexdigest()

    escalator.escalate(report_id=stuck.report_id, tier=stuck.approval_tier,
                       overdue_hours=overdue_hours)
    logger.warning("stuck_escalated", extra=row)
    return row

Because already_paged is checked and updated before the escalation call, the whole sweep is safe to retry: a report that was escalated on a run that then crashed before committing will not be paged a second time, and a report whose stall was cleared and re-created carries a new dedup_key that pages afresh.

Step-by-Step Integration

1. Materialize the open-report set. Query the reports currently in an open approval state and pass their ids into scan_for_stuck; do not pre-filter by age — the sweep decides staleness from the ledger, not from a submitted_at cutoff.

2. Wire the history reader to the immutable ledger. Point history_of at the append-only transition store, never at a mutable updated_at column, so a background touch on the report row cannot reset the stuck clock.

3. Load tier budgets as timedelta values. Convert the business-hours budgets from the Approval SLA Monitoring config into deadlines here; a report whose tier is unknown diverts to review rather than inheriting a default deadline.

4. Persist the dedup set. Back already_paged with a durable store keyed on dedup_key so escalation survives restarts and stays exactly-once per state entry.

5. Verify the detector before enabling paging. Run the sweep against a fixture ledger with a frozen clock and assert the stuck set and its overdue math:

   from datetime import datetime, timedelta, timezone
   
   t0 = datetime(2026, 7, 13, 9, 0, tzinfo=timezone.utc)
   history = {
       "R1": [LedgerEvent("R1", "PENDING_APPROVAL", "manager", t0)],
       "R2": [LedgerEvent("R2", "PENDING_APPROVAL", "manager", t0),
              LedgerEvent("R2", "APPROVED", "manager", t0 + timedelta(hours=2))],
   }
   budgets = {"manager": timedelta(hours=8)}
   frozen = lambda: t0 + timedelta(hours=10)
   
   stuck = list(scan_for_stuck(["R1", "R2"], lambda r: history[r], budgets, frozen))
   assert [s.report_id for s in stuck] == ["R1"]        # R2 already approved
   assert stuck[0].overdue_by == timedelta(hours=2)     # 10h elapsed - 8h budget
   
   paged: set[str] = set()
   class _Esc:
       def escalate(self, **kw): pass
   assert escalate_stuck(stuck[0], _Esc(), paged, frozen) is not None
   assert escalate_stuck(stuck[0], _Esc(), paged, frozen) is None  # exactly once

6. Schedule the sweep and route escalations. Run it on the interval defined by the parent guide and send every escalation into Approval Chain Routing, which re-assigns the stalled report; the detector’s job ends at the audit row.

Edge Cases & Gotchas

FAQ

Why measure time-in-state instead of age since submission?

Age since submission counts time the report spent in states it has already cleared, so a healthy report moving through three reviewers looks as urgent as one truly frozen. A stuck approval is defined by the absence of a state change, not by total age. Measuring from the latest transition into PENDING_APPROVAL isolates the reports that have actually stalled and keeps the alert channel signal-heavy.

How do I avoid paging the same stuck report every interval?

Deduplicate on the specific state entry, not the report id. The scanner builds a dedup_key from the report id and the exact timestamp it entered PENDING_APPROVAL, and escalation records that key in a durable set before signalling. A single stall therefore pages once, while a report that is rejected and re-enters PENDING_APPROVAL later gets a new key and legitimately pages again.

Why read from an audit ledger rather than an updated_at column?

A mutable updated_at moves whenever anything touches the report row — a comment, a background sync, an ERP retry — none of which changes the approval state, so the stuck clock resets and the stall hides. The append-only ledger records only real transitions, so the instant a report entered its current state is immutable and provable. That immutability is also what lets an auditor replay the sweep and get the identical result.

What happens when a report’s approval tier is missing?

The scanner refuses to guess a deadline. If the latest pending transition carries no tier that maps to a budget, the report is skipped from escalation, logged, and diverted to a review lane so a human resolves the misconfigured tier. Assigning a default budget would silently judge the report against the wrong clock and either page too early or never at all.

Related

• Approval SLA Monitoring — the parent guide that defines the budgets, aging buckets, and metrics this sweep consumes
• Approval Chain Routing — receives each escalation and re-assigns the stalled report
• Reimbursement Routing & Approval Sync — the framework that owns the approval ledger this detector reads