Idempotency Fundamentals: Why a Retry Is Only Safe When You Planned for It
A timeout does not tell you the work failed, and if your code assumes it did, your customer gets billed twice.
An operation is idempotent when running it twice leaves the world exactly the way running it once did. That sounds like theory until the night your invoice worker times out, retries, and somebody opens their inbox to two bills with two different numbers on them. The request didn't fail. It just never got around to telling you it worked.
In this post:
- Why a timeout is the most misleading signal in your whole system
- What an idempotency key actually names, and why that distinction is the entire trick
- The two properties that make the pattern hold under real concurrency
- What it costs you, and when you should skip it
1. Why it breaks
Here's the thing nobody tells you when you wire up your first webhook. A network call has three outcomes, not two. It worked, it failed, or you have no idea. That third one is the whole problem, and from where your code is standing it looks identical to the second one.
Your worker posts an invoice to a payment processor. The connection hangs. Twenty five seconds later the HTTP client gives up and raises a timeout. What happened on the other side? Maybe nothing, maybe the request never arrived. Or maybe it arrived, the invoice got created, the customer got emailed, and the only thing that died was the response coming back to you. You cannot tell those apart. Nobody can.
So the code does the reasonable thing and retries. Now there are two invoices.
I'm not the type to get excited about theory for its own sake, but this one earns it, because the obvious fix quietly makes things worse. The obvious fix is check first, then write: query for an existing invoice, only create one if you don't find it. That holds right up until two workers check in the same instant, both see nothing, and both write. You moved the race, you didn't remove it, and honestly that's the worse outcome, because now it fails rarely enough that you'll go blame something else when it finally does.
2. The mechanism
An idempotent operation is one where doing it twice leaves the same result as doing it once. Not "errors out the second time." Same result. The second call is a replay, and the caller can't tell it from the first.
The way you get there is a key, and the key is the part people get wrong. An idempotency key does not name the request. It names the intent. "Invoice job 4471 for the September round" is an intent. It stays the same across every retry, every worker, and every restart, because it comes from what you're trying to do, not from when you happened to try it.
POST /invoice key: inv-4471-sept -> store(key -> result) -> 201 Created
POST /invoice key: inv-4471-sept -> lookup(key) HIT -> 201 (replay)
^
no second invoice is ever written
Same key, same answer. The retry stops at the lookup and never reaches the ledger.
Two properties make that work and both of them matter. The store has to be atomic, so that the check and the claim are one step two workers can't both win. And the result has to be saved, not merely the fact that something happened, so the replay can hand back the same invoice number instead of an empty success that tells the caller nothing.
The HTTP spec's definition of idempotent methods makes a point that surprised me the first time I read it carefully: idempotency is about the effect on the server, not about the response the client sees. Two identical requests can come back with different status codes and still be perfectly idempotent, because the state landed in the same place. If you test for identical responses, you will conclude a correct system is broken.
3. How I run it
My own back office runs on a rule I wrote down after it bit me: anything driven by a webhook, a timer, or an inbound message will fire twice. Not might. Will. Payment processors retry for days when they don't get a fast acknowledgement. Schedulers double fire across a restart. A person clicks submit, sees nothing happen, and clicks again.
So every write that moves money or sends mail goes through the same door. Claim a key, then act. The claim is atomic and it carries a time to live, because a key held forever is its own bug. If the claim succeeds, this worker owns the job. If it doesn't, somebody already owns it and this worker goes home. What surprised me is that once that door existed I stopped thinking about duplicates at all, and the retry logic got simpler, not more complicated, because retrying was finally safe by default.
The failure I keep coming back to is subtler than a double charge. I had a detector watching for an event and a recorder writing it down, and the detector won the race by about a second, so it announced something that hadn't been recorded yet. The fix wasn't a longer wait. It was moving the claim ahead of the write instead of after it, so whoever arrives first owns the entire operation, announcing included. Sequencing is the real product of this pattern. Deduplication is just the part you notice.
4. Tradeoffs
You do pay for it. Every idempotent write needs somewhere durable to keep keys, and that store now sits on the critical path of the very thing you were protecting, which means it can fail too. You have to choose a time to live, and choosing is a judgment call: too short and a slow retry writes the duplicate anyway, too long and a legitimate second request gets swallowed as a replay. Then there's the honest limitation. Idempotency protects you from your own retries, not from a person who genuinely wants two invoices. If the same intent can legitimately happen twice, then the intent isn't what you thought it was, and the key needs another field in it.
Skip it on reads. Skip it anywhere a duplicate is free and nobody would notice. Use it everywhere a duplicate costs money, sends an email, or shows up in front of a customer.
Takeaway
- Treat every timeout as unknown, never as failure.
- Derive the key from the intent, so it survives restarts and racing workers.
- Claim atomically before you act, and store the result so the replay can return it.
A retry is only safe if you decided, in advance, what the second one means.
Jesse