Aivorab
Editorial illustration: Designing Reliable Webhook Delivery for Small APIs: A Conservative Guide

webhooks · api-reliability · event-driven · retry-strategy · idempotency

Designing Reliable Webhook Delivery for Small APIs: A Conservative Guide

A practical guide for indie developers on choosing sync vs async webhook handling, implementing retry with exponential backoff, and using idempotency keys to prevent duplicate event processing.

Published:

The Problem Nobody Talks About

Webhooks seem simple. A provider sends an HTTP POST to your endpoint, you process the event, you return 200. Done. But in production, this simplicity evaporates. Events get lost. Duplicate deliveries pile up. Your endpoint times out and the provider retries until you want to scream. For small teams without dedicated infrastructure, these problems compound quickly.

This guide covers three decisions that separate fragile webhook integrations from reliable ones: synchronous versus asynchronous processing, retry strategy design, and idempotent event handling. The advice here is technically conservative because webhook failures are expensive—lost payments, duplicate shipments, corrupted state.

Synchronous Callbacks vs Asynchronous Queues

The first architectural decision is whether to process webhook payloads immediately or defer them. Most tutorials show synchronous handlers because they’re simpler to write. You receive the request, you do the work, you respond. This works until it doesn’t.

The synchronous trap: Providers time out requests after 5 to 30 seconds depending on the service. Twilio uses 15 seconds for voice webhooks and 5 seconds for Conversations. If your handler takes longer—because you’re calling external APIs, writing to databases, or running business logic—the provider assumes failure and retries. You now have duplicate events piling up while your endpoint is still processing the first one.

The asynchronous pattern: Return a 200 status immediately, then queue the event for background processing. This is the approach recommended by Stripe, Twilio, and multiple engineering teams who learned this the hard way.

// BAD: Synchronous processing
app.post('/webhooks/stripe', async (req, res) => {
  await processPaymentEvent(req.body); // Could take 10+ seconds
  res.status(200).send('OK');
});

// GOOD: Async acknowledgment
app.post('/webhooks/stripe', async (req, res) => {
  await queue.add('process-webhook', req.body);
  res.status(200).send('OK'); // Return immediately
});

The trade-off is operational complexity. Async processing requires a queue system, monitoring for stuck jobs, and a strategy for reprocessing failed events. But synchronous handlers that timeout create a worse problem: silent data loss. As Stigg’s engineering team discovered, the questions you need to answer are: What happens if we’re down and webhooks aren’t processed? What happens if processing fails silently? How do we monitor failures properly? How do we reprocess failed events when we fix bugs? How do we scale without losing data?

Understanding Provider Retry Behavior

Different providers implement retry logic differently. Understanding these differences is essential for designing handlers that work correctly across services.

Stripe retries event delivery for up to 3 days in live mode with exponential backoff. Test mode makes three retry attempts over a few hours. The intervals follow a pattern: 1 hour, 2 hours, 4 hours, 8 hours, and so on. Stripe’s dashboard shows delivery attempts and allows manual retry.

Twilio uses a single retry on timeout by default, but this is configurable via connection overrides up to 5 attempts. Voice webhooks have a 15-second timeout; Conversations webhooks timeout after 5 seconds.

Shopify retries for up to 48 hours with 19 retry attempts. The intervals start at 10 seconds and increase to hours.

GitHub retries up to 3 times within a short window, with failed webhooks visible in the delivery log.

The pattern is clear: providers will retry. Your handler must be prepared for multiple deliveries of the same event. This leads to the second critical decision.

Implementing Exponential Backoff

If you’re building your own webhook sender or retry logic, exponential backoff is the standard approach. The concept is simple: wait progressively longer between retry attempts. This prevents overwhelming the recipient during outages and gives transient failures time to resolve.

A basic implementation looks like this:

async function sendWithRetry(url, payload, maxRetries = 5) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(payload)
      });
      
      if (response.ok) return true;
      
      // Wait before retrying (exponential backoff)
      if (attempt < maxRetries) {
        const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s, 8s, 16s
        await new Promise(resolve => setTimeout(resolve, delay));
      }
    } catch (error) {
      if (attempt === maxRetries) throw error;
      const delay = Math.pow(2, attempt) * 1000;
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

The trade-off is latency versus reliability. Aggressive retries (short intervals, many attempts) can overwhelm a struggling recipient. Conservative retries (long intervals, few attempts) risk losing events during brief outages. Match your retry strategy to your provider’s behavior. If Stripe retries for 3 days, your handler should expect events arriving over that window.

Idempotent Webhook Handling

Duplicate event processing is not a bug—it’s a feature of distributed systems. Networks fail. Providers retry. Your endpoint receives the same event multiple times. Idempotency ensures that processing an event multiple times produces the same result as processing it once.

The golden rule: Key off the event ID, not the payload.

Every major provider includes an event identifier in webhook payloads. Stripe uses event.id. Twilio includes I-Twilio-Idempotency-Token headers. GitHub provides X-GitHub-Delivery headers. Store these identifiers in your database and check them before processing.

const processedEvents = new Set();

app.post('/webhooks', async (req, res) => {
  const eventId = req.headers['x-event-id'] || req.body.id;
  
  if (processedEvents.has(eventId)) {
    console.log('Duplicate event, skipping:', eventId);
    return res.status(200).send('OK');
  }
  
  await processEvent(req.body);
  processedEvents.add(eventId);
  res.status(200).send('OK');
});

For production systems, use a database rather than an in-memory Set. Check for existing event IDs before processing, and mark events as processed after successful handling. This handles restarts, distributed deployments, and long-running processes.

Common idempotency patterns:

  1. Upsert operations: Use INSERT ... ON CONFLICT UPDATE in PostgreSQL, or upsert in MongoDB. This makes repeated processing a no-op after the first successful run.

  2. Status tracking: Store event processing status in your database. Check status before processing, update after completion.

  3. Deduplication tables: Create a separate table for processed event IDs with a unique constraint. Insert the ID first; if it fails due to duplicate, skip processing.

The Transactional Outbox Pattern

For teams building event-driven APIs, the transactional outbox pattern provides a reliable foundation. Instead of sending webhooks directly from application logic, you write events to an outbox table within the same database transaction as your business logic.

-- Universal outbox for all events
CREATE TABLE webhook_outbox (
  id SERIAL PRIMARY KEY,
  event_type VARCHAR(255) NOT NULL,
  aggregate_id VARCHAR(255) NOT NULL,
  payload JSONB NOT NULL,
  status VARCHAR(50) DEFAULT 'pending',
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  processed_at TIMESTAMP
);

The flow works like this:

  1. Application logic executes and writes to the outbox table within the same transaction
  2. A background worker polls for pending events
  3. Worker sends webhooks with retry logic
  4. Worker updates status to ‘processed’ on success, ‘failed’ on permanent failure

This pattern decouples event generation from event delivery. If your application crashes after writing to the outbox but before sending the webhook, the event is preserved. The background worker will eventually process it. This is the approach recommended by the Supabase-WordPress integration discussion for handling webhook reliability.

The trade-off is additional database tables and a background worker. But for small teams, this is cheaper than debugging lost events or duplicate processing in production.

FAQ

Q: Can I just use synchronous handlers if my processing is fast?

A: If your handler consistently completes in under 5 seconds and doesn’t call external services, synchronous processing is acceptable. But “fast today” doesn’t guarantee “fast tomorrow.” As your application grows, synchronous handlers become a liability. Plan for async from the start.

Q: How do I handle webhook signature verification with async processing?

A: Verify signatures before queuing the event. This ensures you’re not processing fraudulent payloads. Stripe, Twilio, and other providers include signing mechanisms (HMAC-SHA1 for Twilio, signature verification for Stripe) that should be checked immediately upon receipt.

Q: What if my queue fills up or workers crash?

A: Monitor queue depth and worker health. Set up alerts for stuck jobs. Implement dead letter queues for events that fail after maximum retries. The GoHighLevel webhook issue highlights that without proper retry modeling, events can be permanently lost.

Q: Should I use a managed webhook service like Hookdeck or AWS SNS?

A: Managed services handle retry logic, delivery tracking, and signature verification for you. They reduce operational burden but add cost and dependency. For small APIs with low event volume, building your own retry and idempotency logic may be simpler and cheaper.

Q: How do I test webhook reliability without a production environment?

A: Use tools like Stripe CLI for local testing. Simulate provider retries by sending duplicate events with the same ID. Test your idempotency logic by processing the same event multiple times. Verify that your handler returns 200 quickly even when background processing is slow.

Bottom Line

Reliable webhook delivery requires three disciplines: return 200 quickly, process events asynchronously, and handle duplicates idempotently. Providers will retry. Your job is to make those retries safe. The transactional outbox pattern adds reliability at the cost of operational complexity. For small teams, this trade-off is usually worth it—lost events are more expensive than background workers.

Start simple. Add async processing and idempotency checks early. Monitor your webhook endpoints. When providers retry and duplicates arrive—and they will—you’ll be ready.

Sources