zlacker

Transactionally Staged Job Drains in Postgres

submitted by johns+(OP) on 2017-09-20 15:21:09 | 115 points 31 comments
[view article] [source] [go to bottom]

NOTE: showing posts with links only show all posts
◧◩
12. brandu+gH[view] [source] [discussion] 2017-09-20 19:32:34
>>njharm+Nv
> Doesn't this just mean bunch of lost jobs when redis fails.

You need to be a little careful about configuration, but believe it or not, as long as Redis is configured with an append only file [1] it will provide you with a durability guarantee just like Postgres would. If it crashes, any jobs that were in there are recovered the next time it starts up.

From the docs:

> From now on, every time Redis receives a command that changes the dataset (e.g. SET) it will append it to the AOF. When you restart Redis it will re-play the AOF to rebuild the state.

---

[1] https://redis.io/topics/persistence

14. heinri+dK[view] [source] 2017-09-20 19:52:35
>>johns+(OP)
Content aside. I never saw a blog article, so carefully typeset as this one:

- Font choices and sizes

- TOC

- Figures

- Code samples

... all look perfect. It even includes a carefully spaced initial.

I'd love to be able to replicate this on my Jekyll blog. But looks like most of this is hand-crafted HTML/CSS: https://github.com/brandur/sorg

◧◩
17. draagl+UL[view] [source] [discussion] 2017-09-20 20:04:21
>>memrac+uI
Bonus: you can use pg_amqp[1] to publish to RabbitMQ with transactional semantics

1: https://github.com/omniti-labs/pg_amqp

◧◩
20. Deimor+QQ[view] [source] [discussion] 2017-09-20 20:39:37
>>bgentr+Kf
> Just be sure to run your enqueueing process as a singleton, or each worker would be redundantly enqueueing lots of jobs. This can be guarded with a session advisory lock or a redis lock.

If you're using PostgreSQL 9.5+ you can also use the new SKIP LOCKED capability, which is perfect for this sort of usage: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...

◧◩◪
22. Deimor+AW[view] [source] [discussion] 2017-09-20 21:17:29
>>netcra+FT
I haven't personally used either of these, but they look interesting and I'm hoping to test them out at some point:

https://github.com/gmr/pgsql-listen-exchange

https://github.com/subzerocloud/pg-amqp-bridge

In theory, these let you use postgres NOTIFY to add messages to queues (which can be done from inside triggers).

[go to top]