Webhook reliability means an event notification can lead to the intended business outcome even when delivery is repeated, delayed or interrupted. A reliable integration checks who sent the message, stores accepted work durably, prevents repeated business actions and gives operators a controlled way to recover failures.
A successful HTTP response is evidence that one request was accepted. It is not proof that the customer account was updated, the order was fulfilled or the support case reached its owner. Those outcomes need their own evidence.
This guide is for technology leaders, product owners and engineers reviewing an integration before release. It proposes an acceptance framework around the journey from notification to completed work, with an illustrative subscription example and a failure rehearsal you can use with an internal team or delivery partner.
Start with the business consequence, not the endpoint
A webhook is a notification sent by one system to another, usually as an HTTP request. Before choosing queues, databases or retry settings, name what the receiving system is allowed to do because of that notification.
Consider a subscription service. A notification may prompt the application to check whether an account should receive access. The business outcome is a valid entitlement with an auditable reason. The message itself is only an input to that decision.
Write a short acceptance statement: which entity changes, which source is authoritative, which transition is allowed and what evidence confirms completion. Include what must never happen twice. Creating two identical delivery requests is a different problem from setting a status field to the same value twice.
This makes the reliability discussion concrete. It also reveals when the integration requirements are still undecided. If nobody can agree what counts as a completed action, revisit software product discovery before treating the endpoint as ready to build.
Read each provider's delivery contract
Do not assume that all webhook providers retry in the same way or retain events for the same period. Record the provider's documented timeout, retry behaviour, identifiers, event versions, verification method and recovery options for the event types you actually use.
Stripe's webhook documentation says deliveries can be duplicated and event order is not guaranteed. It also explains that manual resends can coexist with automatic retries. These are design inputs, not exceptional cases to ignore until an incident occurs.
GitHub's webhook guidance asks receivers to respond within ten seconds and describes asynchronous processing and redelivery of missed deliveries. That limit belongs to GitHub's contract; it is not a universal webhook deadline.
Keep a dated link to each provider's documentation alongside the integration design. Assign someone to review it when subscriptions, API versions or account settings change. The acceptance tests should exercise your actual configuration, including production-like authentication and subscription filters.
Separate receipt, work and outcome
A useful operating model has three stages: receive the notification, perform the work and confirm the business outcome. Give each stage an explicit state and owner. Otherwise a dashboard full of successful deliveries can hide a stalled worker.
For a modest integration, a database-backed inbox and a worker may be enough. A larger system may use a managed queue. The requirement is durable ownership of accepted work, not a particular product or an elaborate architecture.
Receipt: validate before accepting responsibility
Verify the message using the provider's supported mechanism and libraries. For example, Stripe requires the raw request body for signature verification, so middleware must not change those bytes before that check. Scope any secret or account mapping to the expected provider environment.
After verification, validate the event type, required fields and account context. Distinguish an irrelevant but valid event from an event you intended to process but cannot understand. Define the response and operational handling for each case using the provider contract.
Store the accepted event, or an adequate durable work record, before acknowledging receipt. If the process crashes immediately after the response, another process must still be able to find the work. An in-memory task scheduled after returning success leaves a gap at exactly that point.
Work: make processing state visible
Track states such as received, processing, completed, retryable failure and manual review. These are proposed names; choose terms that fit your application. Record attempt history separately so that a failed attempt does not erase evidence of earlier work.
If a worker claims an item and then disappears, the claim needs an expiry or recovery mechanism. A permanent “processing” flag is not a completion guarantee. Define how another worker takes over and how concurrent workers avoid performing conflicting actions.
Outcome: prove the intended change happened
Record a reference to the resulting entitlement, order or case, rather than only a line saying “handler completed”. If the work calls another service, retain its operation reference and enough context to investigate an uncertain response. Keep secrets and unnecessary personal data out of routine logs.
The product owner should be able to ask about a business entity and trace the relevant notification, processing attempts and outcome. That is more useful during support than knowing only that the endpoint returned a successful status code.
Prevent duplicate actions at two different levels
Duplicate delivery and duplicate business work are related but different. The first asks whether you have received this notification before. The second asks whether the intended action has already happened, possibly through another notification or a manual operation.
For delivery tracking, use the provider's stable event or delivery identifier with the appropriate account and environment scope. Enforce uniqueness atomically. A “check then insert” sequence without a database constraint or equivalent coordination can allow two concurrent workers to pass the check.
For business work, choose a key that represents the action's real meaning. In an illustrative entitlement workflow, it could identify an account, a subscription period and the action to grant access. An account identifier alone would be too broad because the same account may legitimately receive later changes.
Do not equate “seen” with “finished”
A deduplication record created at receipt should not cause the worker to skip unfinished work. A repeated delivery may safely acknowledge an already stored item while that item's processing still needs recovery. Keep receipt state and completed outcome distinct.
Likewise, do not mark an external action complete before it succeeds merely to avoid retries. That can convert a visible temporary failure into silently lost work. Record the operation's progress and define what happens at each crash boundary.
Amazon SQS documents at-least-once delivery for standard queues and advises idempotent processing. Adding a queue therefore does not, by itself, remove the need to handle repeated work. Idempotency means repeating an operation does not create an additional unintended effect.
Handle an uncertain external result explicitly
Suppose a downstream service performs an action, but the connection fails before your worker receives the response. Retrying with a new operation identifier could repeat the action. Marking it failed and forgetting it could leave your records inconsistent with the destination.
Where supported, reuse a stable idempotency key according to the destination's contract. Otherwise, query for an operation reference or reconcile the destination state before deciding whether to repeat it. If the destination offers neither a safe retry nor a reliable lookup, route the ambiguity for review instead of promising automatic exactly-once behaviour.
Handle late events without rolling state backwards
Arrival time is not necessarily business sequence. An older notification may reach you after a newer one, and two events can describe different aspects of the same entity. Decide how the application recognises an allowed transition rather than letting the last request overwrite everything.
For a state synchronisation use case, the worker may retrieve the current authoritative object and reconcile the local representation. For a historical ledger, the event's original meaning may need to be retained instead. Fetching the latest object is not an appropriate substitute for every event history requirement.
Make this distinction part of the design review. Define the source of truth, concurrency rule and treatment of obsolete events for each workflow. Where the provider supplies a documented version or sequence, use it within its stated scope. Do not invent global ordering from timestamps.
This is a focused extension of API-first modernisation: a useful integration boundary must define behaviour under failure as well as the shape of a request.
Design replay before you need it
Replay means attempting previously received work again. Reconciliation means comparing the intended business state with the state that actually exists. Both can be necessary: replay cannot recover a notification that your system never received unless the provider supplies a way to retrieve or redeliver it.
Build an operator view that can identify incomplete work by business reference, failure reason, age and attempt count. Define who may trigger a replay, what they must inspect first and how the action is recorded. Start with a narrow selection and bounded rate; an indiscriminate replay can overwhelm a recovering dependency.
A quarantine area for repeatedly failing work is useful only if someone owns it. Give it an alert, a review process and a route back into processing after the cause is understood. Do not allow it to become an invisible list of customer outcomes that never completed.
Set retention around the actual recovery requirement and data-handling policy. Deduplication records that expire before the permitted replay window can allow old actions to happen again. Conversely, keeping every payload indefinitely may retain data the business no longer needs. Document the trade-off and the evidence needed after raw payloads are removed.
A worked example: subscription access after a timeout
Imagine a software business whose application grants access after confirming a subscription's paid status. This is an illustrative design exercise, not a report of a customer implementation.
The receiver verifies a notification, stores it in an inbox and acknowledges it. A worker confirms the relevant subscription state and requests the entitlement using a stable business-operation key. The destination grants access, but a network interruption prevents the response from reaching the worker.
A second delivery now arrives. The receipt uniqueness check identifies the existing inbox item. It does not create a second independent job or falsely declare the original one complete. The worker's recovery path checks the destination using the retained operation reference, or retries with the same supported idempotency key. It then records the existing entitlement as the outcome.
Later, an older notification arrives. The transition rules prevent it from blindly replacing the current entitlement with stale state. A reconciliation check compares the application's entitlement with the authoritative subscription state and flags any unresolved difference.
The acceptance evidence is one correct entitlement, traceable processing attempts and a recoverable audit trail. Merely counting two successful webhook responses would not establish any of those facts.
Run a failure rehearsal before release
Ask the delivery team to demonstrate the following in a controlled environment with approved test data. For each exercise, record the expected outcome, observed result and recovery evidence.
- Repeated delivery: send the same event twice, including concurrently. Confirm one intended business effect and a useful receipt history.
- Crash after receipt: stop processing after durable acceptance. Confirm the work survives and another worker can resume it.
- Uncertain downstream result: interrupt the response after the destination acts. Confirm recovery does not create a second action.
- Reversed arrival order: deliver an older event after a newer one. Confirm the agreed state rule holds.
- Invalid or mismatched request: exercise failed verification and the wrong account context. Confirm no unauthorised work is accepted.
- Unavailable dependency: pause a required service. Confirm bounded retries, visible backlog and recovery without uncontrolled load.
- Missing delivery: omit a notification. Confirm the reconciliation or provider recovery process identifies the affected business entity.
- Operator replay: replay a selected failed item. Confirm access control, audit history and the same duplicate protections used during normal processing.
Choose readiness measures around the service: age of the oldest incomplete item, time from acceptance to business completion, unresolved reconciliation differences and work awaiting manual review. Agree targets with the process owner. A universal latency target would ignore the difference between an urgent access change and an overnight catalogue update.
Make integration readiness a delivery decision
Before approving release, ask for the provider contract, state model, duplicate-action rule, recovery procedure and rehearsal evidence. Name the person responsible for unresolved business outcomes after launch. These artefacts give a reviewer something more concrete than “the webhook works”.
Teams evaluating a delivery partner can add this rehearsal to their engineering partner review. It tests how the team reasons about real operational boundaries without requiring a particular cloud platform or queue vendor.
Start with one consequential workflow and the most damaging plausible duplicate or missed action. Trace it from notification to business result, then test the points where certainty can be lost. If you are planning or reviewing such an integration, contact Vinove with the workflow, participating systems and the outcome you need to protect.
Add to the conversation.
Be the first reader to add a useful perspective.