zlacker

This article glosses over the hardest bit and bike sheds too much over keys.

> Critically, these two things must happen atomically, typically by wrapping them in a database transaction. Either the message gets processed and its idempotency key gets persisted. Or, the transaction gets rolled back and no changes are applied at all.

How do you do that when the processing isn’t persisted to the same database? IE. what if the side effect is outside the transaction?

You can’t atomically rollback the transaction and external side effects.

If you could use a distributed database transaction already, then you don’t need idempotent keys at all. The transaction itself is the guarantee

>>pyroli+(OP)
If you need "transactions" with microservices the traditional answer is sagas - eg multiple transaction boundaries that dont fully commit until their entire set of things is complete by passing a message or event to the next system, and having the ability to rollback each thing in a positive manner, either by appending new correct state again or "not ever adding" the original state.

>>hobs+j2
The problem with sagas is that they only guarantee eventual consistency, which is not always acceptable.

There is also 2 phase commit, which is not without downsides either.

All in all, I think the author made a wrong point that exact-once-processing is somehow easier to solve than exact-once-delivery, while in fact it’s exactly same problem just shaped differently. IDs here are secondary.

>>pyroli+(OP)
The external side-effects also need to support idempotency keys, which you propagate. Then you use something like a message queue to drive the process to completion.

>>pyroli+(OP)
I'm not sure if TFA implies this (it uses too much of his personal jargon for me to understand everything, and it's Friday) but consider this solution based on his transaction log section: you should use the same database that persists the idempotency key to persist the message, and then consume the messages from the CDC/outbox-style. Meaning, the database simply acts as an intermediate machine that dedupes the flow of messages. Assuming you're allowed to make the producer wait.

>>pyroli+(OP)
The practical answer is you use a combination of queries and compensating actions to resemble idempotency with the external service. Some people additionally constrain things to be a linear sequence of actions/effects, and call this pattern Sagas. It's sort of a bastardized distributed transaction that lets you handle a lot of real world use cases without getting into the complexity of true distributed transactions.

>>pyroli+(OP)
i get what you are saying, but i don't think it's fair to call it bike shedding, getting the keys right is also important, one can easily screw up that part too

>>ekropo+65
I'd agree with that - two phase commit has a bunch of nasty failure cases as well, so there's no free lunch no matter what you do when you go distributed. So just ... don't, unless you really really have to.

>>hippo2+26
Exactly that.

>>hobs+hG
You can't avoid distributed unless your users live inside your database.

>>hippo2+26
and when one is a third party service that doesn't?