Introduction

Every engineer who has worked in distributed systems has heard the same advice at some point: “make your operations idempotent.” It gets repeated in design reviews, tacked onto runbooks, and dropped into architecture docs as if the word itself is a solution. But the gap between understanding what idempotency means and actually building systems that enforce it correctly — under real-world failure conditions — is enormous.

Idempotency, at its core, means that performing an operation multiple times produces the same result as performing it once. Simple enough in theory. In practice, the moment you introduce network partitions, concurrent clients, distributed state, caching layers, and multi-step workflows, the definition starts to bend under pressure. “Retry safely” is the starting point — not the destination.

This post goes beyond the basics. We’ll look at how idempotency keys actually work at scale, what deduplication stores look like in production, where the responsibility sits between your API gateway and application code, and — most critically — the failure scenarios that teams routinely overlook until something breaks in production.

If you’re designing payment pipelines, event-driven microservices, or any system where at-least-once delivery is a constraint, this is for you.

Table of Contents

What Idempotency Actually Guarantees (and What It Doesn’t)

Before getting into implementation, it’s worth being precise about the guarantee we’re trying to provide.

An idempotent operation guarantees that the observable state of the system after N executions is the same as after 1 execution. Notice what this does not say: it doesn’t say the operation runs only once, it doesn’t say side effects won’t fire multiple times, and it doesn’t say the response will always be identical.

This distinction matters because teams often conflate idempotency with deduplication. They are related but not the same:

Idempotency is about the outcome — the end state of your system is the same regardless of how many times the operation is applied.

Deduplication is about suppression — detecting and discarding duplicate requests so the operation executes exactly once.

A PUT /users/{id} that sets a user’s email to a specific value is naturally idempotent — calling it ten times leaves the user in the same state. A POST /orders that creates a new order is not naturally idempotent — calling it ten times creates ten orders. To make the latter safe to retry, you need deduplication, and deduplication requires a mechanism to identify what constitutes a “duplicate.”

That mechanism is the idempotency key.

Idempotency Keys: Design and Semantics

An idempotency key is a client-supplied or system-generated identifier that scopes a request to a single logical operation. If the same key is seen again, the system returns the previously computed response rather than re-executing the operation.

Key Properties

Uniqueness scope: An idempotency key must be unique within a meaningful boundary. That boundary is typically: per client + per operation type + per logical intent. A key that is globally unique but reused across different operations by different clients will either cause false deduplication or require complex namespacing logic.

Client ownership: The responsibility for generating idempotency keys almost always belongs to the client. The client knows the intent — it’s retrying because it didn’t receive a response, not because it wants a new operation. Stripe’s API is the canonical example here: clients generate a UUID per “payment attempt intent” and include it in the Idempotency-Key header. The server promises: give me the same key, get the same response.

Time-bounded validity: Keys should not live forever. A reasonable TTL depends on your retry window — if your maximum retry interval is 24 hours, your key store should hold keys for at least that long, typically with some buffer. Stripe holds keys for 24 hours. For most internal service-to-service calls, 1–7 days is sensible depending on the operational context.

Granularity: One key per logical operation, not per HTTP request. If a workflow involves three API calls to complete a single business transaction, you may need idempotency keys at multiple levels — one for each step, and potentially one for the overall saga. Treating each HTTP call as the unit of idempotency leads to correct-but-partial states when retries happen mid-workflow.

Key Generation

UUIDs (v4) are the common default and are generally fine. For systems where you need traceability, a structured key encoding client ID, timestamp, and a random suffix gives you both uniqueness and debuggability. Avoid using business identifiers (order IDs, user IDs) directly as idempotency keys — these are mutable, may not yet exist at request time, and conflate the concept of identity with the concept of intent.

Deduplication Stores: Architecture and Trade-offs

The idempotency key is only as useful as the store backing it. The deduplication store is where you record: “I have seen this key, here is the result.” The design of this store directly determines your system’s correctness under failure.

What the Store Needs to Hold

At minimum, a deduplication record should contain:

  • The idempotency key itself
  • The request fingerprint (a hash of the request body and relevant headers)
  • The response payload or a reference to it
  • The processing status (in-flight, completed, failed)
  • The creation and expiry timestamps

The request fingerprint is often skipped and almost always regretted. Without it, you can’t distinguish between a legitimate retry (same key, same payload) and a client bug (same key, different payload). When payloads differ for the same key, the correct behavior is to return a conflict error — not silently re-apply or deduplicate incorrectly.

Storage Technology Choices

Redis is the most common choice for deduplication stores and for good reason. It supports atomic operations via Lua scripts or transactions, has built-in TTL at the key level, handles high read throughput, and most teams already have it in their stack. The main risks are its memory-first persistence model — if Redis crashes without persistence configured correctly, your deduplication window can have gaps — and the operational complexity of running it at the consistency levels required for financial or critical operations.

Relational databases (PostgreSQL, SQL Server) are underutilized for deduplication and often the better choice for high-stakes operations. A unique constraint on the idempotency key column gives you conflict detection at the database level with no additional logic. You get durability, transactional consistency, and the ability to co-locate the deduplication record with the business data it protects — all in the same transaction. The trade-off is throughput and latency compared to Redis, but for many workloads this is entirely acceptable.

DynamoDB or similar managed NoSQL stores work well for serverless or event-driven architectures where you want managed infrastructure with conditional writes. DynamoDB’s conditional expressions allow atomic “insert if not exists” semantics, which maps cleanly to idempotency key reservation.

The Two-Phase Pattern

The naive deduplication pattern — check if key exists, process if not, store the result — has a fundamental race condition. Two concurrent requests with the same key can both pass the existence check before either has written the result. You end up with double processing.

The correct pattern uses two phases:

Phase 1 — Reservation: Before processing, atomically insert the key with a status of IN_PROGRESS. This must be an atomic “insert if not exists” operation. If the insert fails (key already exists), check the existing status. If it’s COMPLETED, return the stored response. If it’s IN_PROGRESS, you have a concurrent duplicate — return a 409 Conflict with a retry-after hint.

Phase 2 — Completion: After successful processing, update the record to COMPLETED and store the response payload. If processing fails, update to FAILED with the error detail (or delete the record if you want to allow future retries with the same key — this is a policy decision).

This pattern eliminates the check-then-act race condition at the cost of slightly more complex state management. Every production idempotency implementation should use this or an equivalent.

API Gateway vs. Application-Layer Handling

A common architectural question is where idempotency enforcement should live. The answer is: it depends on what guarantee you’re trying to provide, and in sophisticated systems, both layers have a role.

API Gateway-Level Idempotency

API gateways (AWS API Gateway, Kong, Azure API Management, Apigee) can be configured to cache responses keyed on the idempotency key header. For read-heavy or computationally expensive operations, this works well. The gateway intercepts duplicate requests and returns the cached response without the request ever reaching your service.

This approach has real advantages: it offloads deduplication logic from application code, works uniformly across services, and can be applied retroactively without code changes. For simple request-response patterns where the response is self-contained, gateway-level caching covers the common case.

However, gateway-level handling has critical limitations:

It doesn’t protect your data layer. If the gateway crashes after the upstream processed the request but before caching the response, the deduplication record is gone. The next retry reaches the application layer without deduplication, and you get double processing. The gateway can short-circuit round trips, but it cannot be the sole source of truth for idempotency in systems that mutate state.

It conflates caching with idempotency. Gateway response caching is time-bounded and eviction-based. True idempotency requires durable, intentional storage of results. A cached response that expires an hour after a payment was processed is not the same as an idempotency guarantee over a 24-hour retry window.

It doesn’t handle partial failures. If your operation involves multiple downstream services, the gateway sees a single response from your service. It cannot know whether the downstream state is consistent, partially applied, or needs compensation.

Application-Layer Idempotency

Application-layer enforcement owns the deduplication store, executes the two-phase reservation pattern, and has full context about the operation being performed. This is where correctness lives. The application layer can:

  • Validate that the request payload matches the fingerprint for the given key
  • Decide how to handle failed-in-progress records (allow retry vs. return last error)
  • Coordinate idempotency across multi-step workflows
  • Emit exactly-once events downstream as part of the same transaction

The downside is implementation cost — every service needs to implement this correctly, or you need a shared library that enforces the pattern. Inconsistent implementations across services are arguably worse than no implementation, because they create false confidence.

The Right Architecture

In production systems, gateway and application-layer handling are complementary. The gateway handles short-circuit caching for the happy path — a valid retry within the cache TTL gets a fast response without touching the application. The application layer handles correctness — it’s the authoritative source of idempotency state and the only layer that can ensure exactly-once semantics for operations that mutate persistent state.

Think of the gateway as a performance optimization layered on top of correctness guarantees that live in the application.

Failure Scenarios Most Teams Miss

This is where the real complexity lives. The well-known failure cases — network timeouts, transient errors — are well-covered. The following scenarios are less often discussed and more dangerous because of it.

1. The Completed-but-Undelivered Response

This is the most common source of correctness bugs. Your service receives a request, processes it successfully, writes the result — and then the connection drops before the response reaches the client. The client sees a timeout, retries with the same idempotency key, and expects to get the original response back.

The failure mode occurs when the deduplication record is written after the business operation completes but before the response is returned, and the system crashes in between. On retry, there’s no deduplication record, so the operation runs again.

The fix is to write the deduplication record atomically with the business operation — ideally in the same database transaction. If that’s not possible (cross-service, different data stores), then the deduplication record must be written first and the business operation must be idempotent by construction, not by deduplication.

2. The Idempotency Key Reuse Across Different Operations

When clients manage idempotency keys, they occasionally reuse them — through bugs, key generation collisions, or misunderstanding of the semantics. A client reuses a key from a cancelled order for a new order. Your deduplication store has the key, returns the old response, and the client believes a new order was created when it wasn’t.

Mitigation requires the request fingerprint. Hash the request body and critical headers. On key match, compare fingerprints. If they differ, return a 422 or 409 with a clear error message. Log these events — a spike in fingerprint mismatches is a signal of a client bug that needs to be caught early.

3. Concurrent Requests with the Same Key

Two clients (or two threads of the same client) send identical requests simultaneously before either has received a response. Both pass the key existence check. Both proceed to process. You now have a race to write the result.

This is why the two-phase reservation pattern matters. Without atomic reservation, concurrent duplicates result in double processing. Even with reservation, you need to test that your database or cache enforces the uniqueness constraint atomically under concurrent load — this is not a given with all storage technologies or all ORM abstractions.

4. The Partially-Applied Multi-Step Operation

Consider a workflow: debit account → create order → send confirmation email. You wrap this in an idempotency key. On the first attempt, the account is debited, the order is created, and then the service crashes before sending the email. On retry, your idempotency logic sees the key as completed (or in-progress) — what does it do?

If it returns the cached response without re-running the email step, the customer never gets a confirmation. If it re-runs the entire workflow, you debit the account twice. Neither is correct.

The solution is not a single idempotency key over the whole saga, but idempotency at each step, combined with saga/compensation logic. Each step needs its own idempotency key (often derived from the parent key with a step suffix), its own deduplication record, and explicit state tracking. The overall saga must be designed for forward recovery — resuming from the last successful step — not blind retry from the start.

5. Idempotency Keys in Message Queues and Event Streams

Teams often implement idempotency at the API layer and then forget about it entirely when the same operations are triggered via message queues. An SQS queue delivers a message at least once. If your consumer creates an order on each message and the message is delivered twice, idempotency at the API layer offers no protection — the consumer is calling your internal service methods directly.

Every message consumer that performs a state-mutating operation needs its own idempotency mechanism, keyed on a message identifier that survives redelivery. SQS message IDs are not appropriate here — they change on redelivery in some configurations. Embed a business-level correlation ID in the message payload, and use that as your deduplication key in the consumer.

6. TTL Expiry During Active Retry Windows

Your idempotency key TTL expires while the client is still within its retry window. The client retries with the same key, finds no record, and the operation is processed again. For payment systems, this is a potential double-charge scenario.

This is a policy and operations problem as much as a technical one. Your key TTL must be aligned with your client’s maximum retry window, with margin. If your operations team can trigger manual replays of failed events, those replays must be aware of idempotency key semantics and either use the original key (if within TTL) or explicitly override with a new intent.

7. Idempotency Under Schema Evolution

Your service evolves its request schema. A new required field is added. A client still sending the old schema gets a 400, but it’s retrying under an old idempotency key. You later fix the client. Now it retries with the same key, new schema. Your deduplication logic sees the key, compares fingerprints, finds a mismatch, and rejects the request.

This breaks retry flows during deployments and schema migrations. The fix requires versioning your idempotency key namespace alongside your API version, or explicitly invalidating keys when schema-breaking changes are deployed. Neither is trivial, and teams routinely discover this only after a failed deployment.

Cross-Cutting Concerns

Observability

Idempotency is invisible when it works and catastrophic when it doesn’t. Instrument it accordingly. Track:

  • Deduplication hit rate per endpoint (a sudden spike may indicate a client retry storm or a bug)
  • Fingerprint mismatch rate (leading indicator of client bugs)
  • IN_PROGRESS lock age (long-running locks may indicate processing is stuck and blocking legitimate retries)
  • Key TTL proximity alerts for operations still in flight

Testing Strategy

Idempotency is notoriously undertested. Unit tests rarely cover the concurrent duplicate scenario or the crash-between-steps failure. Integration tests should include: concurrent identical requests, retries after simulated failures at each step boundary, and requests that deliberately vary the payload against a reused key. Chaos testing with process kills mid-operation is the only reliable way to validate the crash recovery path.

Downstream Service Idempotency

Your service being idempotent is necessary but not sufficient if it calls downstream services. If your deduplication prevents your service from re-processing, but a downstream call went through on the first attempt and your retry sends it again, you’ve propagated the duplicate outward. Map the idempotency key through your entire call graph. Either ensure each downstream call is also protected by a derived idempotency key, or ensure those calls are naturally idempotent (pure reads, deterministic writes).

Summary

Idempotency done right is a system-wide property, not a feature you add to an endpoint. It requires deliberate decisions about key semantics, durable deduplication stores with the right atomicity guarantees, clear allocation of responsibility between your gateway and application layers, and explicit handling of failure scenarios that go well beyond “retry on 5xx.”

The patterns covered here — two-phase reservation, request fingerprinting, per-step saga keys, consumer-layer deduplication, TTL alignment — are not theoretical. They are lessons extracted from production systems that got hit by exactly the failure scenarios described. None of them are particularly complex in isolation. The difficulty is in applying them consistently, across every state-mutating path in your system, and maintaining them as the system evolves.

“Retry safely” is where you start. Building a system that is actually safe to retry under real-world conditions is the harder, more interesting problem.