There’s no such thing as exactly-once delivery over a network — only at-least-once with idempotent handling on top. Here’s how we designed for it.
Idempotency keys, not idempotent endpoints
We required every mutating request to carry a client-generated idempotency key, stored alongside the resulting response for 24 hours. A retried request with the same key returns the original response instead of re-executing.
What clients actually do
Most client libraries retry on timeout without changing the idempotency key — which is exactly what you want, as long as the server-side lookup happens before any side effect, not after.
The one gotcha
Partial failures — where the side effect happened but the response write failed — are the hard case. We now write the idempotency record and the side effect in the same transaction, so there’s no window where they can disagree.