Zac Sheffer · 2026.9.18 · 7 min read
In a single-tenant system, a webhook arrives and you know who it belongs to. In a multi-tenant system, the first question is which tenant. Getting it wrong is not a bug. It is a data-isolation incident: org A's deal update written into org B's pipeline.
Two failure modes define everything below: resolving the wrong org, and processing the same event twice.
The naive version: the payload carries a connection ID, the connection row names the org. Production disagrees in three ways.
That third point breaks assumptions. A customer has two orgs in Introzy, production and sandbox. They connect the same CRM account to both to test sync behavior against real data. A webhook arrives from that CRM account. The connection ID maps to two rows, one in each org. The payload does not say which org the customer intended because the CRM does not know Introzy has multiple orgs. It just knows it has a webhook URL and something changed. (Stripe's Connect webhook documentation addresses the same class of problem: events must be associated with the correct connected account, and routing depends on metadata rather than the delivery URL alone.)
A worse variant: a customer disconnects their CRM from org A and reconnects it to org B. The old connection row in org A may still exist, soft-deleted or disconnected. If the CRM sends a webhook for an event queued before the disconnect, the connection ID still matches the old row. Routing to org A is wrong. Routing to org B is correct but requires knowing the disconnect-reconnect happened and choosing the active connection.
Most specific rung first:
When the connection ID matches rows in multiple orgs, the latest-updated connection wins. Every candidate is logged for ops.
Our original behavior on ambiguity: return 401. Safe-looking and wrong. It silently dropped real customer events.
What makes the "latest connection wins" heuristic safe: the candidates are rows for the same external connection, orgs that each legitimately connected this account. The worst case is misfiling within that set, never delivery to a stranger. When identity itself is ambiguous (a customer ID mapping to unrelated orgs), we still fail closed. Deterministic beats dead.
Exactly-once delivery is a lie all webhook providers tell politely. The honest contract is at-least-once, and replays are routine. Stripe's webhook documentation notes that providers retry with exponential backoff, sometimes for days. Consumers should design handlers to be idempotent. The standard fix is deduping on the provider's event ID. That assumes the provider sends one. Ours did not.
So the dedup key is built: event type, plus external object ID, plus update timestamp, hashed, held with a 24-hour TTL. Same object, same change, same key. The second delivery is recognized and skipped.
The honest edge case: a payload with no update timestamp degrades the key to type-plus-ID. We log that fallback rather than hide it. The collision risk (two distinct changes to one object arriving without timestamps inside the window) is real, acknowledged, and rare.
Order of operations matters: resolve the org, then dedupe, then enqueue. A dedup check before tenant resolution is a cache keyed on ambiguity.
Two behaviors on the duplicate path. Duplicates return 200 with a duplicate: true body because a 4xx teaches providers to retry harder. And a duplicate still touches the connection's last-webhook timestamp because delivery-health monitoring should reflect that the provider is delivering, even when we decline to process twice.
When a webhook endpoint goes down, the CRM retries. Schedules vary by provider, but the shape is exponential backoff with a cap. Some providers are aggressive. A five-minute outage during a busy sync period can produce hundreds of queued retries, all released in a burst when the endpoint recovers.
We learned this specifically. A deploy took webhook ingress offline for about three minutes. When it came back, we received roughly 450 deliveries in under 30 seconds: a mix of original events and retries.
BullMQ's jobId-based deduplication handled the duplicates. Same event, same dedup key, same jobId. The queue accepted the first and rejected the rest. BullMQ's job ID documentation confirms this: adding a job with an ID that already exists does not create a second job. The non-duplicate events still arrived in a burst. The workers had to absorb them. That meant the workers were already idempotent, the queue had reasonable concurrency limits, and CRM API calls had rate limiting (the CRM's own rate limits enforce it).
The design rule: assume that at any moment, the last five minutes of events might arrive simultaneously. The ingress should accept them all, deduplicate what it can, and let the queue absorb the rest. The worst response to a retry storm is to reject events, because rejection generates more retries. GitHub's webhook best practices make the same point: respond 2xx quickly, defer complex processing, handle failures asynchronously.
Webhooks fail for reasons that are not your problem: malformed payloads, transient network errors, provider schema changes without warning. If you alert on every failed webhook, you will be paged constantly for events that self-heal on retry or require a provider-side fix you cannot influence.
Alert on failed org resolution. A webhook that fails to resolve to an org is either a misconfigured connection (customer action needed), a delivery path the ladder does not handle (your bug), or a data-isolation risk (incident). All three require human attention.
Alert on delivery silence. When a previously-active connection stops receiving webhooks entirely. Silence is more concerning than a noisy error because it means something changed on the provider side and no retry will fix it. We track last-webhook timestamps per connection and alert when a previously-active connection goes quiet longer than its expected delivery interval.
Everything else (individual job failures, transient errors, malformed payloads) goes to structured logs with org and connection context. Queryable, not alerting. Reviewed during business hours.
If the dedup store is down, we enqueue anyway. We decided this upfront, not after an incident. The reasoning: a duplicate event is recoverable downstream where upserts and stable job IDs absorb it. A dropped event is a customer's deal change that silently never happened. Under partial failure, prefer the error you can detect and recover from.
Ingress hands a resolved, deduped job to the queue. The worker completes it. Permanent conditions return structured skips instead of throwing: sync disabled, org paused, stage unmapped. Retrying a permanent condition wastes attempts. Skips emit a UI event so the interface says "skipped, here is why" instead of looking stuck. Only transient failures throw. User-facing failure events fire only on the final retry because a status light that flaps on every transient error is a status light nobody believes.
Outbound jobs mirror the discipline: stable job IDs derived from the entity so a burst of edits coalesces in the queue instead of racing through it. The outbound half of the architecture covers that side.
Webhooks are what sits under every sync promise the product makes.
Introzy is free to start. No card required.