Skip to content

Webhooks between systems: what the receiving side must do before go-live

7 min read
Custom Software
Webhooks between systems: what the receiving side must do before go-live

Key Takeaways

  • 1Verify the signature on the raw request body before you act on any webhook
  • 2Acknowledge fast, queue the work, and process it in the background
  • 3Assume every event can arrive twice and record event IDs with the result
  • 4Run a scheduled comparison against the provider API to catch lost events

A webhook looks like a small feature until a customer is charged once and fulfilled twice, or an order never reaches your warehouse system and nobody notices for a week. The sending service promises to tell you when something happens. Whether your systems stay correct depends on the receiving side, which your team or an integration supplier builds.

This guide is for technology and product leaders who connect their own software to payments, an online shop, a CRM or GitHub. It covers four things a receiver must do before production, what three major providers document about them, and what to ask whoever builds the integration.

Verify the signature before anything else

A webhook endpoint is a public URL that triggers business actions. Without a check, anyone who finds the address can post a fake "payment succeeded" message. Stripe's documentation names this risk: an attacker could send fake events that fulfil orders, grant access or modify records.

Each provider signs the request with a shared secret, and your code recomputes the signature and compares. Three details cause most failures:

  • Use the raw body. Stripe and Shopify both warn that a framework which parses the JSON first changes the bytes, so the signature never matches.
  • Compare in constant time. GitHub tells receivers to avoid a plain equality operator to mitigate timing attacks.
  • Check the timestamp where the provider signs one. Stripe's libraries default to a five-minute tolerance, and Stripe warns that a tolerance of zero disables the recency check.

Keep the secret in a secrets store, never in the repository, as GitHub advises. Reject failures with an error status and do no further work on them. A receiver that skips verification should not go live, whatever the deadline says.

Acknowledge fast and queue the work

Providers treat a slow answer as a failed delivery. GitHub expects a 2XX response within 10 seconds, and Shopify uses a one-second connection timeout and five seconds for the whole request. Stripe gives no number but says the endpoint must return a success code before any complex logic that could time out.

The pattern is simple. The endpoint verifies the signature, stores the raw event in a queue or database table, and returns 200. A separate worker does the real work: updating the invoice, creating the order, notifying the customer. GitHub and Shopify both recommend a queue, and Shopify adds that it absorbs bursts of traffic.

The table of received events also gives support an answer to "did the shop tell us about this order?" An endpoint that works inline will time out under load, and the provider then resends the event while the first attempt is still running. Reply first and work later, and keep a record of what arrived.

Assume every event can arrive twice

Stripe notes that endpoints might occasionally receive the same event more than once and recommends logging processed event IDs. Shopify says your app might receive the same webhook more than once, for example after a network timeout, and offers the X-Shopify-Webhook-Id header for deduplication. GitHub recommends checking the X-GitHub-Delivery header to protect against replay attacks.

Two practical points go beyond the documentation and are our own judgement:

  1. Record the ID together with the effect. If you save the event ID on arrival and the worker then crashes, the retry is skipped and the order is lost. Write the ID in the same database transaction as the business change.
  2. Know what the ID identifies. GitHub states that a redelivery keeps the same delivery ID, so an event you never completed can be replayed only if you did not mark it done. Stripe adds that two separate event objects can describe the same change, so for money also check the object ID and event type.

Do not rely on order either. Stripe does not guarantee delivery in the order events were generated. A handler that needs the latest state should fetch the object from the API instead of trusting the payload. The safe receiver treats each event as repeatable and out of order.

Close the gaps with a scheduled comparison

Retries help, but they stop. The documented limits show how long a gap you can survive:

Provider Signature header Reply window After failures
Stripe Stripe-Signature "Quickly", no figure Automatic retries for up to three days in live mode
GitHub X-Hub-Signature-256 10 seconds No automatic redelivery; manual or API redelivery within 3 days
Shopify X-Shopify-Hmac-SHA256 5 seconds in total 8 retries over 4 hours, then an Admin API subscription is deleted

Read the last column as a deadline. If your receiver is down over a long weekend, a GitHub event from Friday may be outside the redelivery window by Monday, and a Shopify subscription may be gone.

The fix is a scheduled reconciliation job. Shopify describes it: a job that periodically retrieves data you might have missed. Stripe likewise points to using the API to retrieve missing objects. The job asks the provider which payments, orders or pushes exist for a period, compares them with your records, and sends any missing ones through the same worker as normal events.

The schedule is our judgement, not a documented rule. A nightly run suits most businesses and keeps the gap far inside the windows above; run it more often where a missing event costs money. Alert a person whenever the job finds anything, because a missing event means the live path has a fault.

When this is more than you need

A notification that posts to a team chat, or any event whose loss costs nothing, can run with a signature check, a quick reply and a log. Where the provider offers no API to list past events, skip the comparison and watch failed deliveries in its dashboard. Retry counts and time limits come from current provider pages and change, so re-read them when you design the receiver. We found no independent data on how often duplicates or lost events occur, so treat the four controls as insurance, not a response to a known failure rate. Where webhooks move money, stock or access, build all four.

What to ask a supplier before go-live

A short written answer to each question separates careful work from optimistic work:

  1. Which signature is verified, and on which bytes? The answer names the provider's header and says the raw body is used.
  2. How fast does the endpoint reply, and where is the work done? Look for a queue or a table of received events.
  3. What is the deduplication key, and where is it stored? Look for the provider's event or delivery ID, written with the business change.
  4. What happens if the receiver is down for a day? Look for a reconciliation job, its schedule and who is alerted.
  5. Who owns the signing secret, and how is it rotated? Shopify notes a rotated secret can take up to an hour to apply.

If you build webhooks for your own customers, the open Standard Webhooks specification is a yardstick: it signs the message ID, timestamp and body together, treats the ID as an idempotency key and recommends retries over multiple days. It is a proposal, not a law.

Put the five answers on one page before the integration goes live and revisit them after the first outage. Our articles on six questions before integrating two systems and API integration patterns for growing companies cover the design around the receiver, and the post-incident review template helps when an event does go missing. If you want a second opinion on a receiver you are about to ship, see our custom software work.

Sources

Share this article

Interested in Custom Software & Systems?

Learn more about how we can help your organization with this service.

Explore Service

Need expert guidance?

Let's discuss how our experience can help solve your biggest challenges.