Aivorab

idempotency · API design · retries · timeouts · small teams · distributed systems

Idempotency Keys and Retries: A Practical Guide for Small APIs

Learn how to implement idempotency keys, handle retries, and set timeouts without overengineering your public API. A conservative approach for indie developers and small teams.

Published:

The Problem: Duplicate Charges and 3 AM Pages

You’ve shipped a payment endpoint. A user clicks “Pay” twice, or their mobile network hiccups, and suddenly they’ve been charged twice. Your logs show two successful POST requests 847 milliseconds apart. Customer support is already drafting a refund email. This isn’t a race condition—it’s Tuesday.

For small public APIs, the fix isn’t more code; it’s better contracts. You need idempotency keys for mutating operations, a retry strategy with exponential backoff, and sensible timeouts. The goal is to make your API safe to retry without turning your codebase into a distributed systems textbook.

What Is an Idempotency Key?

An idempotency key is a unique identifier (usually a UUID v4) that the client attaches to a request to guarantee that performing the same operation multiple times produces the same result only once. The server remembers the key and returns the cached response instead of re‑executing the operation.

In HTTP, some methods are naturally idempotent: GET, PUT, and DELETE. POST and PATCH are not. That means every POST that creates a resource (payments, orders, subscriptions) must be made safe through an explicit idempotency mechanism.

Why Retries Happen—and Why They Matter

Networks are unreliable. Clients time out, mobile networks drop packets, and infrastructure hiccups. When a client doesn’t receive a response, it retries. Without idempotency, each retry can trigger a duplicate side effect: a double charge, an extra inventory deduction, a duplicate record.

A 2024 survey of 400+ backend teams found that 73% had shipped idempotency logic that failed in production under real mobile network conditions. The culprit isn’t missing documentation—it’s that most teams implement idempotency keys the way they’d implement any other feature: store a hash, check it, done.

The Common Mistake: Storing Keys Without Locking State

Here’s the pattern you’ll see in production code reviews:

  1. Client sends POST /payments with Idempotency-Key: abc123.
  2. Server checks if abc123 exists in storage.
  3. If not, it processes the payment and stores the key.
  4. If yes, it returns the cached response.

This looks clean. It’s stateless. And it’s wrong.

The failure happens when the client retries before step 4 completes. A network hiccup, a Lambda cold start, a slow database query—doesn’t matter. Two requests with abc123 arrive simultaneously. Both pass the “Have I seen this?” check. Both charge the card.

You’re not just checking if a key exists. You’re checking if processing has started, is in‑flight, or finished. A boolean flag can’t capture that. A timestamp can’t either. You need an atomic lock that prevents concurrent execution while tracking the outcome.

How to Implement Correctly: Atomic Locks and State Tracking

A robust implementation treats the idempotency key as a state machine with three states: pending, success, and failure.

When a request arrives:

  1. Look up the key.
  2. If it’s success or failure, return the cached response immediately.
  3. If it’s pending or absent, acquire an atomic lock (e.g., a Redis SET NX or a database row lock) and transition the state to pending.
  4. Execute the operation.
  5. On success, store the response and mark the key success.
  6. On failure, mark the key failure and return an error.
  7. Release the lock.

This ensures that only one request with a given key is processed at a time. Concurrent retries will wait for the lock, then return the cached result.

Trade‑offs: Simplicity vs. Correctness

You don’t always need a full state machine. The right level of complexity depends on your traffic and risk.

Keep it simple if:

Invest in correctness if:

A middle ground is to store the key with a status and a response, but use a database unique constraint to prevent concurrent processing. If two requests arrive simultaneously, the second will fail the unique constraint and you can return a 409 Conflict. However, as discussed in Hacker News, a 409 tells the client nothing about whether the original request succeeded or failed. The client must then decide whether to retry or ask the user. For most small APIs, that’s an acceptable trade‑off—if you document it clearly.

Practical Steps for a Small Public API

  1. Identify mutating endpoints. Every POST, PATCH, or DELETE that creates or changes state should support idempotency keys.
  2. Require the header. Use Idempotency-Key (or Idempotency-Request-Id) in the request header. Reject requests without it for critical endpoints.
  3. Choose storage. Redis with TTL is fast and simple. A relational database with a unique index and a status column works too. Pick what you already use.
  4. Set an expiration. Keys should not live forever. 24 hours is a common default; adjust based on your business logic.
  5. Handle concurrent requests. Use an atomic lock (Redis SET NX, database advisory locks, or optimistic concurrency) to ensure only one request per key is processed at a time.
  6. Implement client‑side retries. Advise your API consumers to use exponential backoff with jitter. Never retry idempotent methods (GET, PUT, DELETE) without a key—they’re already safe.
  7. Set timeouts. Define clear timeouts for your API (e.g., 5 seconds for most operations, 30 seconds for heavy computations). Timeouts prevent long‑running requests from holding locks indefinitely.
  8. Document everything. Tell developers how to use idempotency keys, what headers to send, what responses to expect, and how retries should work. A well‑written API doc is your first line of defense.

FAQ

Q: Do I need idempotency for GET requests? A: No. GET is idempotent by definition—calling it multiple times has the same effect as calling it once. However, you should still consider caching to avoid duplicate work.

Q: What if the client sends a different payload with the same idempotency key? A: That’s a client error. Your API should reject the request with a 400 Bad Request or 422 Unprocessable Entity. The key must map to a single, specific operation.

Q: How long should I store idempotency keys? A: As long as it takes for a client to reasonably retry. 24 hours covers most mobile network scenarios. Longer retention increases storage costs and privacy risk; shorter retention risks duplicates.

Q: Can I use a database unique constraint instead of Redis? A: Yes. A unique constraint on the idempotency key column prevents duplicate inserts. Combine it with a status column and a transaction to track state. This is simpler and avoids an extra dependency, but may be slower under high concurrency.

Q: Should I return 409 Conflict or replay the success response on a duplicate key? A: It depends on your threat model. If you want to force the client to handle conflicts explicitly, return 409. If you want to hide complexity and ensure the client gets a consistent response, replay the cached success. Most large APIs replay success; small APIs can choose either, but document the behavior clearly.

Conclusion

Idempotency isn’t a luxury—it’s a necessity for any API that handles retries. For small teams, the goal is to implement it correctly without overengineering. Use atomic locks, set sensible timeouts, store keys with expiration, and document your choices. Your users (and your on‑call schedule) will thank you.

Sources