Reliable Webhook Handling for Solo SaaS: Retries, Idempotency, and Debugging

Webhooks are how most SaaS products learn about money, user changes, and external events. Stripe tells you a payment succeeded. GitHub tells you a push happened. Your own background jobs may notify another service. The promise is simple: when something interesting occurs, your endpoint gets an HTTP POST.

The reality is messier. Providers retry when they do not receive a success response. Networks drop packets. Your process restarts mid-handler. The same event can arrive twice, or arrive after you already acted on it. Solo founders often discover this when a customer is charged twice, an email is sent three times, or a subscription status flips back and forth.

This is an operating guide for handling webhooks reliably when you are the only engineer.

Treat delivery as at-least-once

Most webhook providers guarantee at-least-once delivery, not exactly-once. They retry on timeouts, 5xx responses, and sometimes on 4xx responses. Retry windows differ: some last hours, others days. The exact schedule is in the provider’s documentation; do not assume a universal default.

Because retries exist, your handler must be safe to run more times than the provider intended. The classic failure is a handler that creates a record or charges a card, then the response is lost, and the provider retries. Without protection, you create a second record.

Acknowledge quickly, then do the work

The first rule is operational: return a 2xx response as soon as you have verified the request and recorded enough to process it later. Many providers time out after a few seconds. If your handler calls external APIs, sends email, or runs heavy logic before responding, you will trigger unnecessary retries even when the work succeeded.

A durable pattern:

1. Verify the signature (or equivalent authentication).
2. Extract the provider’s stable event ID.
3. Persist the raw event or a reference to it (or enqueue it).
4. Return 200 (or 202) immediately.
5. Process the business logic asynchronously.

If you cannot enqueue yet, at least keep the synchronous path short and idempotent. Long-running work in the request thread is how solo founders end up with retry storms during a slow database migration or a temporary third-party outage.

Make the handler idempotent with the provider’s event ID

Idempotency means processing the same event one time or one hundred times produces the same final state. The practical way is to deduplicate on the provider’s event identifier.

Examples of identifiers (illustrative; always check the current docs for the provider you use):

• Stripe: event.id (starts with evt_)
• GitHub: X-GitHub-Delivery header
• Shopify: X-Shopify-Webhook-Id
• Standard Webhooks / many modern systems: webhook-id header

Store the event ID in a table with a unique constraint before you perform side effects. A simple schema looks like:

CREATE TABLE processed_webhook_events (
  event_id TEXT PRIMARY KEY,
  provider TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ
);

The critical detail is atomicity. A check-then-insert done in two steps can race: two concurrent retries both see “not present” and both proceed. Use the database’s unique constraint (or an equivalent atomic operation) as the gate. If the insert fails because the ID already exists, return 200 and stop. Do not return an error for a duplicate; an error tells the provider to retry again.

Inside the handler, prefer upserts and conditional updates over blind inserts. When you call downstream APIs that support idempotency keys, pass a stable key derived from the event ID so their side effects are also safe.

Verify signatures and protect against replays

Anyone who discovers your webhook URL can send traffic to it. Always verify the signature or HMAC using the provider’s secret and the raw request body. Do this before you store or enqueue the event.

Most providers include a timestamp in the signature. Reject requests whose timestamp is too old (a common window is a few minutes). This limits replay attacks if a signed payload is captured. Store and rotate secrets carefully; never commit them to the repository.

If a provider offers multiple signing secrets during rotation, support both until the old one is retired.

Handle failures and poison events

Not every event should be retried forever. Distinguish transient failures (database blip, temporary rate limit) from permanent ones (invalid payload, business rule violation you cannot recover from).

For transient failures, let the provider retry or implement your own bounded retry with exponential backoff and jitter if you control the queue. For permanent failures, record the event as failed, alert yourself, and stop retrying it. Otherwise a bad payload can consume your retry budget or fill your logs indefinitely.

When you process asynchronously, make the worker itself idempotent. A job that crashes after updating the database but before marking the event processed should be safe to run again.

Observability that fits one person

You need to answer: did this event arrive, was it processed, and what happened?

At minimum:

• Log the event ID, provider, event type, and outcome (accepted, duplicate, processed, failed).
• Include a correlation or request ID that ties the webhook to any downstream work.
• Alert on a rising rate of signature failures, unprocessed events older than a threshold, or repeated failures of the same event type.
• Keep a simple way to re-deliver or manually reprocess an event (provider dashboard, or a small admin action that re-enqueues by ID).

You do not need a full distributed tracing platform on day one. Structured logs with the event ID plus an error tracker for exceptions are usually enough to debug the next customer complaint about a missing update.

A practical checklist

Before you consider a webhook endpoint production-ready:

• Signature verification on the raw body using the current secret.
• Timestamp tolerance to limit replays.
• Unique constraint on the provider event ID.
• Atomic deduplication (unique constraint or single transaction).
• Fast 2xx response; heavy work moved to a queue or background job.
• Idempotent business logic (upserts, conditional writes, downstream idempotency keys).
• Bounded retries and a path for permanent failures.
• Logs that include the event ID and outcome.
• A way to inspect and reprocess a single event.
• Secrets stored outside the codebase and rotatable.

Common solo-founder pitfalls

• Doing the full business logic before returning 200, then wondering why the provider keeps retrying.
• Using a hash of the body as the deduplication key when the provider already supplies a stable ID (payloads can differ slightly across attempts).
• Returning 4xx or 5xx on duplicates, which causes more retries.
• Logging the full payload (including secrets or customer data) by default.
• Assuming “it only fires once” because it did so in testing.

When the complexity is justified

Not every webhook needs a durable queue on day one. If volume is tiny and the handler is a few database writes that are already idempotent, a synchronous path with the unique-constraint pattern may be sufficient. Add a queue when timeouts appear, when processing calls slow external services, or when you need better visibility into failures.

The goal is not architectural purity. It is that the same event cannot charge a customer twice, create duplicate records, or leave your system in an inconsistent state—even when the network is imperfect and you are the only person watching the logs.

Webhooks are one of the places where “it worked in development” is least predictive of production behavior. Build the idempotency and the fast path early; they are cheaper than reconstructing what happened after a customer reports a double charge.