zlacker

You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.

>>really+(OP)
What happens when you need to retry the messages from a few minutes in the past because there was a transient failure in a downstream dependency?

>>really+(OP)
You reminded me of a post on Dropbox announcement in 2007, that you can do it “yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem”.

Just because you can, doesn’t mean you should.

>>really+(OP)
Sounds really painful since I want things like retry logic, retry queues, and dead letter queues.

>>pbourk+z
S3 went down twice in 5 years. Since we're transferring files, you just push everything in the next window. The retry is trivial from the agent and accounted for in the consumer.

>>Hatche+K
Cost is the motivating factor here.

>>really+Bt
Dropbox is more expensive than an FTP server, so the two scenarios are comparable.

>>really+rt
I wasn’t talking about the reliability of S3, but of your own systems.

Say the outage results in a few million messages that need to be retried. Some subset of those few million will never succeed (aka they are “poisoned pills”). At the same time, new messages are arriving.

In your system, how do you maintain QoS for incoming messages as well as allow for the resolution of the few million retries while also preventing the poisoned pills from blocking the queue? How do you implement exponential backoff, which is the standard approach for this?

SQS gives you some simple yet powerful primitives such as the visibility timeout setting to address this scenario in a straightforward manner.

>>really+(OP)
Which gets you one of the basic features of SQS, but not the entire rest of the implementation. It's also significantly more work than just setting up an SQS queue.

I guess if you're at the point where your engineering time to implement this + all of the features on top of it that you might need from SQS and future maintenance of this custom solution is cheaper than the cost of using SQS, and you have no other outstanding work that your engineering team should be doing instead, this is a viable cost optimization strategy.

But that's a whole lot of ifs, and with customers I've mostly worked with, they're far better served just using SQS.