16 - Idempotency and Exactly-Once Delivery

Why Idempotency Matters

Networks are unreliable. Requests time out, clients retry, load balancers re-route, and message queues redeliver. Any system that communicates over a network must assume that a single logical operation may arrive more than once.

Without idempotency, duplicates cause real damage:

A user gets charged twice for the same purchase
An order is placed twice, shipping two packages
A notification email is sent three times
A counter increments on every retry, inflating metrics

An idempotent operation is one where performing it multiple times produces the same result as performing it once. The system's state after N identical requests is identical to the state after one request.

Naturally idempotent operations:

GET /users/42 — reading never changes state
PUT /users/42 { name: "Alice" } — full replacement overwrites to the same value
DELETE /orders/99 — deleting an already-deleted resource is a no-op

NOT naturally idempotent:

POST /orders — each call creates a new order
UPDATE balance SET amount = amount + 10 — each call increments
INSERT INTO logs (...) — each call appends a new row

The goal is to make non-idempotent operations behave idempotently through explicit design.

Idempotency Keys

The most common pattern for achieving idempotency in APIs is the idempotency key:

The client generates a unique key per logical operation (typically a UUID)
The client sends this key in a header: Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
The server checks if it has seen this key before
If yes → return the cached result without re-executing
If no → execute the operation, store the result keyed by the idempotency key, return the result

Storage options for idempotency keys:

Redis with TTL — SET idempotency:{key} {result} NX EX 86400 (24-hour expiry)
- NX = set only if the key does not already exist (atomic check-and-set)
- EX 86400 = auto-expire the key after 86,400 seconds (24 hours)
Database table — unique constraint on the key column, store the response

Server-side flow:

receive request with idempotency_key
  → lookup key in store
  → if found:
      return stored result (same status code, same body)
  → if not found:
      execute business logic
      store result mapped to key
      return result to client

Important considerations:

Keys should expire (24–72 hours) to avoid unbounded storage growth
The store must be checked atomically — race conditions between concurrent retries can cause double-execution
If the first request is still in-flight when a retry arrives, return 409 Conflict or 429 Retry Later

At-Least-Once vs At-Most-Once vs Exactly-Once

Message delivery guarantees define how systems handle failures in asynchronous communication:

At-most-once delivery:

Send the message and never retry
The message may be lost, but it will never be duplicated
Example: UDP packets, fire-and-forget logging
Use when: losing a message is acceptable, but duplicates are not

At-least-once delivery:

Retry until the receiver acknowledges
The message will definitely arrive, but it may arrive multiple times
Example: SQS, RabbitMQ, Kafka (default), most webhook systems
Use when: you cannot afford to lose messages

Exactly-once delivery:

Each message is processed exactly one time — no loss, no duplicates
True exactly-once at the network level is impossible (Two Generals Problem)
Practical exactly-once is achieved by combining at-least-once delivery with an idempotent consumer

Exactly-once processing = At-least-once delivery + Idempotent consumer

The delivery system guarantees the message arrives (retrying as needed). The consumer guarantees that processing it again has no additional effect.

Deduplication Strategies

Multiple techniques exist for making consumers idempotent:

1. Database unique constraint

Add a UNIQUE constraint on (idempotency_key) or (message_id)
Attempt the insert — if it violates the constraint, the message was already processed
Simple, durable, works with any relational database

2. Redis SET NX (atomic check-and-set)

SET msg:{message_id} 1 NX EX 3600
If the SET succeeds → process the message
If it fails (key exists) → skip, already processed
Fast, but data is lost if Redis restarts without persistence

3. Message-level deduplication

Maintain a set of seen message IDs (in-memory, Redis, or database)
Before processing: check if message_id is in the seen-set
After processing: add message_id to the seen-set
Prune old entries with TTL or sliding window

4. Transactional outbox pattern

Write the business result AND mark the message as processed in the same database transaction
If the transaction commits → both succeed atomically
If it rolls back → neither persists, message will be redelivered and retried
Eliminates the gap between "processed" and "acknowledged"

Real-World Examples

Payment processing (Stripe):

Client sends Idempotency-Key header with the charge request
If the network drops the response and the client retries, Stripe returns the original charge result
Without this: customer is charged twice, refund required, trust damaged

Order creation (e-commerce):

Client generates order_id before submitting
Server uses order_id as the idempotency key with a unique constraint
Second submission returns the existing order instead of creating a duplicate

Webhook delivery (GitHub, Stripe):

Each webhook event has a unique event_id
The receiver stores processed event_id values
On redelivery (sender didn't get 2xx), receiver checks the ID and skips if already handled

Queue consumers (SQS, Kafka):

Consumer pulls message → processes it → sends acknowledgment
If the consumer crashes after processing but before acknowledging, the message is redelivered
The consumer must be idempotent: check if the work was already done before doing it again
Pattern: use the message's unique ID as a deduplication key in your database

Key Takeaways

An idempotent operation produces the same result regardless of how many times it is executed
Idempotency keys let clients safely retry without causing duplicate side effects
True exactly-once delivery is a myth — practical exactly-once means at-least-once delivery paired with idempotent processing
Use atomic check-and-set (Redis NX or DB unique constraints) to prevent race conditions in deduplication
The transactional outbox pattern ensures processing and acknowledgment are atomic
Design every write operation assuming it will be called more than once — because eventually, it will be