zlacker

FWIW there is a queue based on maildir which has implementations in Perl, Python, C and Java and probably more.

The Perl implementation was the original AFAIK.

http://search.cpan.org/dist/Directory-Queue/

>>emmela+(OP)
Interesting; looks like DirectoryQueue uses directories, rather than file locks (man 2 flock), to lock the queue messages. This might actually work, since mkdir returns an error if you attempt to create a directory that already exists. The implementation seems to be handling most of the obvious failure cases, or at least tries to.

https://metacpan.org/release/Directory-Queue/source/lib/Dire...

So how does one lock a message in s3? Does s3 have a "createIfDoesNotExistOrError"? I'm still having difficulty understanding how the proposed system avoids race conditions.

>>emmela+(OP)
Back in 2000, I worked with a guy that built an entire message queue product using the SMTP protocol with an implementation that was in turn built on top of lex and yacc.

>>archgo+AC
maildir is specified at https://cr.yp.to/proto/maildir.html and billions of message uses it each year. So that's pretty safe.

I can't vouch for the queueing code but I believe it's quite robust too.