>>bo0tzz+(OP)
What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue?
Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
>>hipade+Xb
I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
>>zackki+yd
No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data.
We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
>>eximiu+3g
So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
>>iskela+yT
Either business data and job are committed or none of them. Then as you write, either polling or listening to an even worker, can pick it up.
Bonus stuff, from implementation perspective, is that if worker selects row FOR UPDATE (locking the job from others to pick up) and dies, Postgres will release the lock after some time, making the job available for other workers.