Build a Reliable Webhook Consumer with Idempotency, Retries, and Signature Verification

What Problem This Solves

A webhook endpoint often starts as a small HTTP route:

1. Receive a request.
2. Parse the JSON body.
3. Update your database.
4. Return 200 OK.

That implementation works in a local demo and fails under normal delivery conditions.

Webhook providers commonly retry when your endpoint times out, returns an error, or becomes temporarily unreachable. Network intermediaries can also repeat requests. A provider may deliver an event more than once, deliver events out of order, or send an event that your application cannot process until a related record exists.

A reliable consumer treats webhook delivery as an at-least-once message system. The practical consequences are important:

• A request may arrive more than once.
• A successful HTTP response does not mean business processing completed.
• A request body must be verified before trusting its contents.
• Slow work should not happen while the provider is waiting for an HTTP response.
• Every event needs an observable lifecycle.

This tutorial implements a small Node.js service backed by PostgreSQL. The design is intentionally conventional: verify the raw request body, persist the event, acknowledge it quickly, and process it asynchronously with an idempotency constraint.

The example uses a generic HMAC signature header named x-webhook-signature. Adapt the header name and signing format to the provider you integrate with. The architecture remains the same even when the provider uses a timestamped signature or a different digest algorithm.

Prerequisites

You need:

• Node.js with support for modern JavaScript syntax.
• PostgreSQL.
• A webhook signing secret.
• Basic familiarity with HTTP, SQL, and asynchronous JavaScript.
• A way to send test HTTP requests, such as curl.

Create a project and install the application dependencies:

mkdir webhook-consumer
cd webhook-consumer
npm init -y
npm install express pg dotenv

The example uses CommonJS modules to keep the setup small. Add a start script to package.json:

{
  "scripts": {
    "start": "node src/server.js"
  }
}

Create a .env file for local development:

PORT=3000
DATABASE_URL=postgres://postgres:postgres@localhost:5432/webhooks
WEBHOOK_SECRET=replace-with-a-long-random-secret

Do not commit this file. In a deployed environment, inject these values through the platform’s secret-management mechanism.

Design the Event Lifecycle

Before writing the route, define the state transitions. A useful minimal lifecycle is:

• received: the signature was valid and the event was stored.
• processing: a worker has claimed the event.
• processed: business handling completed successfully.
• failed: processing failed and the event can be retried.
• dead_letter: repeated attempts exceeded the retry limit.

The initial HTTP request should perform only work required to authenticate and durably record the event. It should not send emails, call several downstream APIs, or perform expensive database operations unrelated to event storage.

A PostgreSQL table can enforce the most important property: one provider event ID is stored only once.

CREATE TABLE webhook_events (
  id BIGSERIAL PRIMARY KEY,
  provider_event_id TEXT NOT NULL UNIQUE,
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'received',
  attempts INTEGER NOT NULL DEFAULT 0,
  available_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  locked_at TIMESTAMPTZ,
  processed_at TIMESTAMPTZ,
  last_error TEXT,
  received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  CHECK (status IN ('received', 'processing', 'processed', 'failed', 'dead_letter'))
);

CREATE INDEX webhook_events_ready_idx
  ON webhook_events (available_at, received_at)
  WHERE status IN ('received', 'failed');

The unique constraint is more reliable than an application-level check followed by an insert. Two requests can pass a SELECT check at the same time. The database constraint closes that race.

The available_at column lets a worker postpone retryable failures without blocking newer events.

Verify the Raw Request Body

Signature verification must use the exact bytes sent by the provider. Parsing JSON and serializing it again can change whitespace, escaping, or property ordering. Those changes can produce a different HMAC even though the parsed object looks equivalent.

Configure Express so the webhook route receives the raw body. Keep this route before a global JSON parser, or use a route-specific raw parser.

const express = require('express');
const crypto = require('node:crypto');
const { Pool } = require('pg');
require('dotenv').config();

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

function signaturesMatch(rawBody, suppliedSignature, secret) {
  if (typeof suppliedSignature !== 'string' || suppliedSignature.length === 0) {
    return false;
  }

  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');

  const supplied = Buffer.from(suppliedSignature, 'utf8');
  const calculated = Buffer.from(expected, 'utf8');

  if (supplied.length !== calculated.length) {
    return false;
  }

  return crypto.timingSafeEqual(supplied, calculated);
}

The length check matters because timingSafeEqual requires buffers of equal length. The comparison should happen only after checking that the header exists.

Now add the endpoint:

app.post(
  '/webhooks/provider',
  express.raw({ type: 'application/json', limit: '1mb' }),
  async (req, res) => {
    const signature = req.get('x-webhook-signature');

    if (!signaturesMatch(req.body, signature, process.env.WEBHOOK_SECRET)) {
      return res.status(401).send('invalid signature');
    }

    let event;

    try {
      event = JSON.parse(req.body.toString('utf8'));
    } catch (error) {
      return res.status(400).send('invalid JSON');
    }

    if (
      !event ||
      typeof event.id !== 'string' ||
      typeof event.type !== 'string' ||
      !event.data
    ) {
      return res.status(400).send('invalid event shape');
    }

    try {
      const result = await pool.query(
        `INSERT INTO webhook_events
          (provider_event_id, event_type, payload)
         VALUES ($1, $2, $3)
         ON CONFLICT (provider_event_id) DO NOTHING
         RETURNING id`,
        [event.id, event.type, event]
      );

      if (result.rowCount === 0) {
        // The event was already durably accepted.
        return res.status(200).send('already accepted');
      }

      return res.status(202).send('accepted');
    } catch (error) {
      console.error('webhook persistence failed', error);
      return res.status(500).send('temporary failure');
    }
  }
);

The endpoint returns 202 Accepted for a newly stored event because processing is asynchronous. Returning 200 for a duplicate is also appropriate: the duplicate has already been accepted and should not cause unnecessary provider retries.

A database outage returns a server error. That tells a provider with retry support to try again later. Do not return success when durable storage failed, or the event may be lost.

Add the normal JSON parser only after the webhook route:

app.use(express.json({ limit: '1mb' }));

app.get('/health', async (req, res) => {
  try {
    await pool.query('SELECT 1');
    res.status(200).json({ ok: true });
  } catch (error) {
    res.status(503).json({ ok: false });
  }
});

app.listen(process.env.PORT || 3000, () => {
  console.log(`listening on port ${process.env.PORT || 3000}`);
});

The ordering is significant. If express.json() runs first, req.body will be an object rather than the original byte buffer, and the signature check will no longer verify the provider’s exact message.

Add a Transactional Worker

Persisting an event is only half the system. A worker must claim ready events, process them, and update their status.

The claim query should use a row lock so multiple worker processes do not handle the same event concurrently. PostgreSQL’s FOR UPDATE SKIP LOCKED is useful for a queue-like table because workers skip rows currently claimed by another worker.

async function claimEvent(client) {
  const result = await client.query(`
    WITH next_event AS (
      SELECT id
      FROM webhook_events
      WHERE status IN ('received', 'failed')
        AND available_at <= NOW()
        AND attempts < 5
      ORDER BY received_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1
    )
    UPDATE webhook_events AS event
    SET status = 'processing',
        attempts = attempts + 1,
        locked_at = NOW(),
        updated_at = NOW()
    FROM next_event
    WHERE event.id = next_event.id
    RETURNING event.*
  `);

  return result.rows[0] || null;
}

A simple processing loop can run in the same Node.js process for a small service:

function retryDelay(attempts) {
  const seconds = Math.min(300, 2 ** attempts);
  return `${seconds} seconds`;
}

async function handleEvent(event) {
  switch (event.event_type) {
    case 'customer.created':
      // Perform an idempotent application operation here.
      return;
    case 'customer.deleted':
      // Perform the corresponding deletion or deactivation.
      return;
    default:
      throw new Error(`unsupported event type: ${event.event_type}`);
  }
}

async function processOneEvent() {
  const client = await pool.connect();
  let event;

  try {
    await client.query('BEGIN');
    event = await claimEvent(client);
    await client.query('COMMIT');

    if (!event) {
      return false;
    }

    try {
      await handleEvent(event);

      await client.query(
        `UPDATE webhook_events
         SET status = 'processed',
             processed_at = NOW(),
             updated_at = NOW(),
             locked_at = NULL
         WHERE id = $1`,
        [event.id]
      );
    } catch (error) {
      const nextStatus = event.attempts >= 5 ? 'dead_letter' : 'failed';

      await client.query(
        `UPDATE webhook_events
         SET status = $1,
             available_at = NOW() + ($2)::interval,
             last_error = $3,
             updated_at = NOW(),
             locked_at = NULL
         WHERE id = $4`,
        [nextStatus, retryDelay(event.attempts), String(error.message), event.id]
      );
    }

    return true;
  } catch (error) {
    await client.query('ROLLBACK').catch(() => {});
    console.error('worker failure', error);
    return false;
  } finally {
    client.release();
  }
}

setInterval(async () => {
  try {
    await processOneEvent();
  } catch (error) {
    console.error('worker loop failure', error);
  }
}, 1000);

This example is intentionally small, but there is an important limitation: the worker marks an event as processing before calling handleEvent. If the process crashes during handling, the event can remain stuck in processing.

In production, add a lease timeout. A recovery query can move old processing rows back to failed when locked_at is older than a chosen threshold. The handler itself must still be idempotent because a crash can occur after the business action succeeds but before the event is marked processed.

For business operations, use a second idempotency key where possible. For example, if a webhook creates an internal invoice, store the provider event ID or provider object ID on the invoice table with a unique constraint. Queue-level deduplication protects event handling; business-level deduplication protects the side effect.

Test the Complete Path

Generate a signature from the exact request body. The body sent with curl must be identical to the body used to calculate the signature.

BODY='{"id":"evt_1001","type":"customer.created","data":{"customer_id":"cus_42"}}'
SIGNATURE=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex | awk '{print $2}')

curl -i \
  -X POST http://localhost:3000/webhooks/provider \
  -H 'content-type: application/json' \
  -H "x-webhook-signature: $SIGNATURE" \
  --data "$BODY"

The first request should be accepted. Sending the same request again should return an accepted response without inserting a second database row.

Verify the database state:

SELECT
  provider_event_id,
  event_type,
  status,
  attempts,
  last_error,
  received_at,
  processed_at
FROM webhook_events
ORDER BY received_at DESC;

Test at least these cases:

• A valid event.
• The same valid event delivered twice.
• A modified body with the old signature.
• A missing signature header.
• Malformed JSON with a valid signature for that malformed byte sequence.
• A database outage during insertion.
• A handler failure followed by a retry.
• A worker restart while an event is being processed.

Common Errors

Parsing JSON before verifying the signature

This is the most common integration mistake. The signature is calculated over bytes, not over an abstract JavaScript object. Install the raw-body route before the JSON middleware and verify req.body before parsing it.

Using a normal equality comparison

A direct string comparison is simple but does not provide the same timing behavior as a constant-time comparison. Use the platform’s timing-safe comparison function and handle unequal lengths first.

Treating an event ID as globally unique without checking the provider

Some providers guarantee globally unique event IDs. Others scope identifiers to an account, endpoint, or object type. Confirm the provider’s contract. If IDs are scoped, make the database constraint composite, such as (account_id, provider_event_id).

Returning success before durable storage

An in-memory queue, log statement, or scheduled promise is not durable acceptance. If the process exits after the response but before persistence, the event may disappear. Store the event before acknowledging it.

Performing slow work in the HTTP request

Webhook providers have delivery timeouts. A slow downstream API call can cause a retry even when the first request eventually succeeds, creating duplicate work. Acknowledge after persistence and process separately.

Assuming database deduplication makes side effects safe

The database prevents duplicate event rows, but it cannot undo an external email, payment request, or API call. Make downstream operations idempotent, use provider-supported idempotency keys, or record an outbox operation in the same database transaction as the state change.

Leaving processing rows permanently locked

Worker crashes are normal operational events. Add lease recovery, monitor the age of processing rows, and expose metrics for failed and dead-letter events.

Ignoring event ordering

At-least-once delivery does not automatically provide ordering. If customer.deleted arrives before customer.created, your handler needs a policy: defer the event, fetch current state from the provider, or treat the event as a reconciliation signal rather than a complete source of truth.

Operational Checklist

Before production, verify the following:

• Signature verification uses the raw request bytes.
• The signing secret is stored outside source control.
• The event table has a uniqueness constraint.
• Duplicate requests return a successful response after the original event is stored.
• The HTTP handler does not perform slow business work.
• Events have bounded retries and a dead-letter state.
• Stuck processing rows can be recovered.
• Business side effects are idempotent.
• Payloads do not accidentally retain credentials or unnecessary personal data.
• Logs include the provider event ID and internal event ID.
• Metrics cover accepted, duplicate, invalid, failed, processed, and dead-letter events.
• A replay procedure exists for dead-letter events.

A replay command should be explicit and auditable. Do not silently reset every failed event during deployment. Operators should be able to select an event, inspect its error, and replay it after correcting the underlying problem.

FAQ

Should the endpoint return 200 or 202?

Either can work if the behavior is consistent with the provider’s expectations. 202 Accepted clearly communicates that the event was stored for asynchronous processing. For duplicate deliveries, 200 or 202 is reasonable because the event is already accepted. The critical rule is to return a success status only after durable storage has succeeded.

Do I need a message broker?

Not necessarily. A PostgreSQL-backed queue is often enough for modest traffic and keeps the system operationally simple. A dedicated broker becomes more attractive when you need high throughput, independent consumer scaling, built-in delivery semantics, or many separate consumers. Do not add one solely to solve duplicate delivery; idempotency is required with either design.

What should happen when an event cannot be processed?

Retry transient failures with a bounded backoff. Move permanent failures or exhausted retries to a dead-letter state. Store the last error and preserve the original payload so an operator can diagnose and replay the event. Avoid retrying validation errors forever.

How should webhook payloads be stored?

Store the fields needed for processing and debugging, but review privacy and retention requirements. JSON storage is convenient for provider-specific payloads. For frequently queried business fields, copy them into typed columns or application tables rather than relying on deeply nested JSON queries everywhere.

Can I process events concurrently?

Yes, but concurrency changes the correctness requirements. Use row locking or a broker’s claiming mechanism, enforce idempotency at the business boundary, and consider per-resource ordering when events for the same customer, order, or subscription must be serialized. Start with a measured worker limit instead of unbounded promises.

Is signature verification enough for security?

No. Signature verification proves that the request was produced with the shared secret, assuming the secret is protected. You still need request-size limits, TLS, secret rotation procedures, replay protection when supported by the provider, authorization checks for the affected tenant, careful logging, and validation of the event schema.

Related Reading

• When an AI Agent Makes a Mistake in Production, Which Layer Should Stop It?
• Build an AI Trust Harness: Model-Agnostic Evals, Tracing, and Release Gates for LLM Apps
• Practical Automated Feature Engineering in Python: Validating py-evoFE for Tabular ML