It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution.
Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.)
Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps.
https://getoban.pro/articles/one-million-jobs-a-minute-with-...
No, we don't operate like that. Call me out when I'm wrong technically, but don't tell me that because someone is some sort of celebrity that I should cut them some slack.
Everything he pointed out is literally covered in the GCP Tasks documentation.
Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these same problems. I wrote a bit about them here in our docs: https://riverqueue.com/docs/transactional-enqueueing
Ultimately I want to help engineers be able to focus their time on building a reliable product, not chasing down distributed systems edge cases. I think most people underestimate just how far you can get with this model—most systems will never outgrow the scaling constraints and the rest are generally better off not worrying about these problems until they truly need to.
Please check out the website and docs for more info. We have a lot more coming but first we want to iron out the API design with the community and get some feedback on what features people are most excited for. https://riverqueue.com/
We've also had a lot of experience with with other libraries like Que ( https://github.com/que-rb/que ) and Sidekiq (https://sidekiq.org/) which have certainly influenced us over the years.
They’re surprisingly easy to implement in plain SQL:
[1] https://taylor.town/pg-task
The nice thing about this implementation is that you can query within the same transaction window
Just skimming the docs, can you add a job directly via the DB? So a native trigger could add a job in? Or does it have to go via a client?
I'd be curious to compare performances once you guys are comfortable with that, we do them openly and everyday on: https://github.com/windmill-labs/windmill/tree/benchmarks
I wasn't aware of the skip B-tree splits and the REINDEX CONCURRENTLY tricks. But curious what do you index in your jobs that use those. We mostly rely on the tag/queue_name (which has a small cardinality), scheduled_for, and running boolean which don't seem good fit for b-trees.
* neoq: https://github.com/acaloiaro/neoq
* gue: https://github.com/vgarvardt/gue
Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out.
Gue has been around for a while and is probably serving its users well.
Looking forward to trying out River now. I do wonder if neoq and river might be better off joining forces.
Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poolers like pgbouncer, which prevent NOTIFY ever triggering, and the extra db load of trigger handling wasn't worth it.
[1]: https://github.com/sorentwo/oban/commit/7688651446a76d766f39...
I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed previously by the community in posts such as these [1][2][3]
I am assuming that bgentry and brandur have strong thoughts on the topic since they avoided the default Go license choice of BSD/MIT, so I'd love to hear more.
[1] https://www.makeworld.space/2021/01/lgpl_go.html [2] https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-... [3] https://softwareengineering.stackexchange.com/questions/1790...
Starting with the project's tagline, "Robust job processing in Elixir", let's see what else:
- The same job states, including the British spelling for `cancelled`
- Snoozing and cancelling jobs inline
- The prioritization system
- Tracking where jobs were attempted in an attempted_by column
- Storing a list of errors inline on the job
- The same check constraints and the same compound indexes
- Almost the entire table schema, really
- Unique jobs with the exact same option names
- Table-backed leadership election
I'll try to work this into the higher level docs website later today with an example :)
- Use FOR NO KEY UPDATE instead of FOR UPDATE so you don't block inserts into tables with a foreign key relationship with the job table. [1]
- We parallelize worker by tenant_id but process a single tenant sequentially. I didn't see anything in the docs about that use case; might be worth some design time.
[1]: https://www.migops.com/blog/select-for-update-and-its-behavi...
Unlike https://github.com/tembo-io/pgmq a project we've been working on at Tembo, many queue projects still require you to run and manage a process external to the database, like a background worker. Or they ship as a client library and live in your application, which will limit the languages you can chose to work with. PGMQ is a pure SQL API, so any language that can connect to Postgres can use it.
While there is no overlap in technology or structure with Sidekiq, the original Oban announcement on the ElixirForum mentions it along with all of the direct influences:
https://elixirforum.com/t/oban-reliable-and-observable-job-p...
a job queue might just be the tip of the use cases iceberg... isn't it?
in the end it's a pub/sub - I use nats.io workers for this.
arf, just read a few comments on this same line down bellow.
One solution is the outbox pattern:
https://microservices.io/patterns/data/transactional-outbox....