zlacker

Idempotency keys for exactly-once processing

submitted by defly+(OP) on 2025-12-01 12:07:33 | 182 points 67 comments
[view article] [source] [go to bottom]

NOTE: showing posts with links only show all posts
23. otterl+sdg[view] [source] 2025-12-05 23:23:24
>>defly+(OP)
This is some useful reading that's in the same vein: https://docs.aws.amazon.com/wellarchitected/latest/reliabili...
42. AdieuT+iPg[view] [source] 2025-12-06 06:04:27
>>defly+(OP)
From the article:

  In distributed systems, there’s a common understanding that 
  it is not possible to guarantee exactly-once delivery of 
  messages.
This is not only a common understanding, it is a provably correct axiom. For a detailed discussion regarding the concepts involved, see the "two general's problem"[0].

To guarantee exactly once processing requires a Single Point of Truth (SPoT) enforcing uniqueness shared by all consumers, such as a transactional persistent store. Any independently derived or generated "idempotency keys" cannot provide the same guarantee.

The author goes on to discuss using the PostgreSQL transaction log to create "idempotency keys", which is a specialization of the aforementioned SPoT approach. A more performant variation of this approach is the "hi/low" algorithm[1], which can reduce SPoT allocation of a unique "hi value" to 1 in 2,147,483,648 times when both are 32-bit signed integers having only positive values.

Still and all, none of the above establishes logical message uniqueness. This is a trait of the problem domain, in that whether two or more messages having the same content are considered distinct (thus mandating different "idempotentcy keys") or duplicates (thus mandating identical "idempotency keys").

0 - https://en.wikipedia.org/wiki/Two_Generals'_Problem

1 - https://en.wikipedia.org/wiki/Hi/Lo_algorithm

◧◩
50. gunnar+84h[view] [source] [discussion] 2025-12-06 09:50:01
>>attila+Deg
"Idempotency key" is a widely accepted term [1] for this concept; arguably, you could call it "Deduplication key" instead, but I think this ship has sailed.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

◧◩◪◨⬒⬓⬔
55. ekjhgk+2lh[view] [source] [discussion] 2025-12-06 13:09:14
>>vouwfi+Efh
> in a distributed system of transactions where availability is guaranteed, performance is evaluated horizontally, change is frequent and easy,

Isn't that the situation inside a CPU across its multiple cores? Data is replicated (into caches) in a distributed system of transactions, because each core uses its own L2 cache with which it interacts, and has to be sent back to main memory for consistence. Works amazing.

Another even more complex system: a multi CPU motherboard supporting NUMA access: 2 CPUs coordinate their multiple cores to send over RAM from the other CPU. I have one of these "distributed systems" at home, works amazing.

[1] https://en.wikipedia.org/wiki/Non-uniform_memory_access

[go to top]