One approval engine, 78 kinds of approval
Approval chains as configuration rather than code, and the race I left in it that someone else had to find.
Published 2026-10-15 by Arif Hossen
Tags: Operations
Source: https://arifhossen.dev/blog/one-approval-engine-78-kinds-of-approval
Where it started
At an East African insurer, approvals had grown one screen at a time. Finance had its own approval code, HR had its own, and the tables that set prices had none: anyone with edit rights could change a rate, a discount or a product, with no second approver and no record.
I replaced the per-screen logic with one engine, in a single change across 120 files. By the time I left it gated 78 types of record, from vendor bills and payroll to claim payments and reserves.
The order of signatures is configuration
Chains as data
A chain is a list of rows: an approval type, an approver, a position, and whether that approver is mandatory. Administrators edit it in the admin panel. Changing who signs a vendor bill is a form, not a deployment.
A record finalises only when every mandatory approver on its chain has approved. Each step records who, when and a comment, and the history is copied onto the record itself, so the audit trail survives outside the approvals table. Rejection finalises too, with a reason.
Maker-checker where auditors look
Set-up data
Payments were already controlled. The gap was the reference data that decides them: charges and levies, fire and marine rates, discounts, products, policy covers, the bank list, service providers, vehicle makes.
Each of those gained an approval status and joined the engine as its own type. An unapproved chart-of-accounts entry cannot be posted to. An unapproved rate cannot price a policy. Adding a type is an enum case and a chain, not a project.
Approvers are functions, not role names
Roles
The system runs in several environments, and each names its roles differently. Code that sends to HR by role label silently finds nobody on a deployment where HR is called something else.
The engine reasons about functions instead: HR, director, super-approver, admin. A config file maps each function to the real role on that deployment, and a notifier falls back to configured recipients when the role is empty. A reminder that reaches nobody is now a configuration error, not a quiet miss.
The race I left in it
What I missed
processApproval() records the approval, checks whether the record is fully approved, and finalises it, all inside one transaction. For a payroll payment, finalising means creating journal entries and posting to the ledger. That is slow, so the transaction stays open a long time.
Under MySQL's default REPEATABLE READ isolation, a second approver acting at the same moment reads a snapshot from before the first commit. Their check misses the other approval, so the record never finalises, even though every required approver has signed. It stays pending until someone edits the chain.
I did not find this. The engineer who took the system over did, after I left. Their fix re-checks payroll payments after commit, in a fresh read, and reconciles any record a chain change leaves stuck.
Slow work inside a transaction widens every race
What it cost
The design was sound with one approver and wrong under concurrency, and none of my tests ran two approvers at once. What I do differently now: anything heavy that follows a decision runs after the commit that records it, and any check that decides whether something is finished gets a test with two writers.