HoopoTrack: multi-tenant SaaS with metered, idempotent notifications

Repair shops lose a surprising amount of their day to one question: is my item ready yet? HoopoTrack answers it without a phone call. Staff move a ticket through a workflow they define; the customer follows a public tracking link with a live timeline, and a screen in the waiting room shows the current queue.

That is the product in a sentence. The engineering interest is almost entirely in what happens when a ticket changes status.

The constraint

A status change can fan out to email and WhatsApp. Both cost money, both are metered per plan, and both are delivered by a queue worker that can be retried at any time — after a timeout, after a deploy, after a crash mid-send.

That combination produces three failure modes that all matter commercially:

• Double-send. The customer gets the same “your repair is ready” message twice. Cheap to cause, expensive in trust.
• Phantom quota burn. A send is counted against the shop’s allowance but never actually goes out, or goes out twice and is counted once.
• Lost delivery on rollback. The job is dispatched, the surrounding transaction then rolls back, and the worker picks up a job referring to a status that no longer exists.

The naive version — dispatch a job from a model observer and decrement a counter inside the job — fails all three.

The decision

Delivery is modelled as a row, not an action.

When a ticket status changes, a NotificationDelivery record is created inside the same database transaction as the status change itself, with a unique constraint on the pair of (ticket status, channel). Creating the row is the claim to send. Two concurrent attempts race for the same unique key; the loser catches the constraint violation, re-reads the winning row, and does nothing.

The quota is reserved at the same moment, against the same transaction — before anything is dispatched. If the reservation fails because the shop is out of allowance, the whole thing rolls back together and no delivery row exists to be retried.

Only then is the send job queued, and it is queued afterCommit. A rollback therefore cannot leave a job pointing at a status that was never persisted.

Failures are then split into two kinds, explicitly, as distinct exception types:

• Transient — a timeout, a 5xx from the provider, a rate limit. Worth retrying; the delivery row stays pending and the reserved quota stays reserved.
• Terminal — an invalid phone number, a template rejected by the provider, a hard 4xx. Not worth retrying; the row is marked failed and the reservation is released.

Collapsing these two into a generic “it failed, retry it” is how you get a queue that spends all night retrying a phone number that will never be valid, while the shop’s allowance stays locked up.

What it cost

This is more machinery than “send an email when the status changes”, and it is worth being honest about the bill:

• An extra table and a unique index on the hot path of the most common write in the product.
• A reconciliation command. Rows can still get stuck — a worker dies between reserving and sending, and no exception is ever thrown to classify. A scheduled job sweeps for deliveries left pending beyond a threshold and re-queues or fails them. Idempotency is what makes that safe to run.
• Harder tests. The interesting cases are concurrent ones, so the suite includes tests that hammer the quota manager from parallel processes rather than asserting on a single happy path.

The payoff is that the difficult questions have boring answers. Can a customer get two messages? No, the unique constraint prevents it. Can a shop be billed for a message that never sent? No, terminal failures release the reservation. What happens if we deploy mid-send? The job is retried and finds the work already claimed.

The rest of the surface

The notification pipeline is the part worth writing about, but the product is a full SaaS and the supporting work is most of the code:

• Tenancy as a trait plus policies, with owner, staff, customer and platform-admin roles, and per-staff ability flags rather than coarse role checks.
• Quota and metering across tickets, workflows, clients, staff and each message channel, with monthly allowances, hard caps, and purchasable extra credits.
• Subscription lifecycle driven by webhook listeners — payment success and failure, refunds, entitlement sync on login — with in-app banners when a payment fails rather than a silent downgrade.
• Customer-facing surface hardening: rate-limited public routes, signed URLs for photos, optional PIN gates on tracking pages, and a dedicated test group for public endpoint protection.
• Trilingual, including right-to-left Arabic — around a thousand translation keys per locale across the app, plus locale-prefixed marketing routes.
• An audit log that captures tenant and user context, changed values, IP and user agent, surfaced in-app rather than left in a table nobody reads.