zlacker

[parent] [thread] 22 comments
1. really+(OP)[view] [source] 2019-05-27 08:07:23
Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis).

> Oh but what about ORDERED queues? The only way to get ordered application of writes is to perform them one after the other.

This is another WTF. Talking about ordered queues is like talking about databases, because it's data that's structured. If you can feed data from concurrent sources of unordered data to a system where access can be ordered, you have access to a sorted data. You deal with out-of-order data either in the insertions or a window in the processing or in the consumers. "Write in order" is not a requirement, but an option. Talking about technical subjects on twitter always results in some mind-numbingly idiotic statements for the sake of 144 characters.

replies(3): >>obitua+z >>cldell+A >>archgo+O
2. obitua+z[view] [source] 2019-05-27 08:17:21
>>really+(OP)
In the first subheading, towards the bottom, the author mentions it’s $0.40 per million api calls.
3. cldell+A[view] [source] 2019-05-27 08:17:33
>>really+(OP)
S3's eventual consistency aside, how is it cheaper?

It would seem to me that naively, S3 charges $5 per million POST requests, so it's 10x worse than SQS's $0.40 per million.

replies(1): >>really+x1
4. archgo+O[view] [source] 2019-05-27 08:20:00
>>really+(OP)
> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags.

Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

replies(2): >>really+o1 >>emmela+U6
◧◩
5. really+o1[view] [source] [discussion] 2019-05-27 08:26:55
>>archgo+O
You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.
replies(4): >>pbourk+X1 >>Hatche+82 >>static+1r >>cthalu+0d1
◧◩
6. really+x1[view] [source] [discussion] 2019-05-27 08:28:05
>>cldell+A
Writing directly from EC2 is free (within same region).
replies(1): >>etaioi+N1
◧◩◪
7. etaioi+N1[view] [source] [discussion] 2019-05-27 08:31:05
>>really+x1
Only the data transfer is free. The API requests are not free. S3 does come out to be more expensive...
replies(1): >>really+Ku
◧◩◪
8. pbourk+X1[view] [source] [discussion] 2019-05-27 08:33:15
>>really+o1
What happens when you need to retry the messages from a few minutes in the past because there was a transient failure in a downstream dependency?
replies(1): >>really+Pu
◧◩◪
9. Hatche+82[view] [source] [discussion] 2019-05-27 08:34:31
>>really+o1
You reminded me of a post on Dropbox announcement in 2007, that you can do it “yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem”.

Just because you can, doesn’t mean you should.

replies(1): >>really+Zu
◧◩
10. emmela+U6[view] [source] [discussion] 2019-05-27 09:42:18
>>archgo+O
FWIW there is a queue based on maildir which has implementations in Perl, Python, C and Java and probably more.

The Perl implementation was the original AFAIK.

http://search.cpan.org/dist/Directory-Queue/

replies(2): >>archgo+uJ >>coredo+z21
◧◩◪
11. static+1r[view] [source] [discussion] 2019-05-27 13:22:53
>>really+o1
Sounds really painful since I want things like retry logic, retry queues, and dead letter queues.
◧◩◪◨
12. really+Ku[view] [source] [discussion] 2019-05-27 13:54:01
>>etaioi+N1
Only if you are pushing 1 message per write. 1 write per agent per minute, ends up at just over 2$ per month fox 10 agents, regardless of the number of messages - it's cheaper for any setup I've encountered. AWS services are always middling solutions, which you can often optimize for better cost efficiency.
replies(2): >>etaioi+Bg1 >>kondro+Kl2
◧◩◪◨
13. really+Pu[view] [source] [discussion] 2019-05-27 13:55:10
>>pbourk+X1
S3 went down twice in 5 years. Since we're transferring files, you just push everything in the next window. The retry is trivial from the agent and accounted for in the consumer.
replies(1): >>pbourk+ZX
◧◩◪◨
14. really+Zu[view] [source] [discussion] 2019-05-27 13:56:00
>>Hatche+82
Cost is the motivating factor here.
replies(1): >>earenn+wy
◧◩◪◨⬒
15. earenn+wy[view] [source] [discussion] 2019-05-27 14:28:59
>>really+Zu
Dropbox is more expensive than an FTP server, so the two scenarios are comparable.
◧◩◪
16. archgo+uJ[view] [source] [discussion] 2019-05-27 16:07:54
>>emmela+U6
Interesting; looks like DirectoryQueue uses directories, rather than file locks (man 2 flock), to lock the queue messages. This might actually work, since mkdir returns an error if you attempt to create a directory that already exists. The implementation seems to be handling most of the obvious failure cases, or at least tries to.

https://metacpan.org/release/Directory-Queue/source/lib/Dire...

So how does one lock a message in s3? Does s3 have a "createIfDoesNotExistOrError"? I'm still having difficulty understanding how the proposed system avoids race conditions.

replies(1): >>emmela+ED1
◧◩◪◨⬒
17. pbourk+ZX[view] [source] [discussion] 2019-05-27 17:58:18
>>really+Pu
I wasn’t talking about the reliability of S3, but of your own systems.

Say the outage results in a few million messages that need to be retried. Some subset of those few million will never succeed (aka they are “poisoned pills”). At the same time, new messages are arriving.

In your system, how do you maintain QoS for incoming messages as well as allow for the resolution of the few million retries while also preventing the poisoned pills from blocking the queue? How do you implement exponential backoff, which is the standard approach for this?

SQS gives you some simple yet powerful primitives such as the visibility timeout setting to address this scenario in a straightforward manner.

◧◩◪
18. coredo+z21[view] [source] [discussion] 2019-05-27 18:39:21
>>emmela+U6
Back in 2000, I worked with a guy that built an entire message queue product using the SMTP protocol with an implementation that was in turn built on top of lex and yacc.
◧◩◪
19. cthalu+0d1[view] [source] [discussion] 2019-05-27 20:12:43
>>really+o1
Which gets you one of the basic features of SQS, but not the entire rest of the implementation. It's also significantly more work than just setting up an SQS queue.

I guess if you're at the point where your engineering time to implement this + all of the features on top of it that you might need from SQS and future maintenance of this custom solution is cheaper than the cost of using SQS, and you have no other outstanding work that your engineering team should be doing instead, this is a viable cost optimization strategy.

But that's a whole lot of ifs, and with customers I've mostly worked with, they're far better served just using SQS.

◧◩◪◨⬒
20. etaioi+Bg1[view] [source] [discussion] 2019-05-27 20:53:12
>>really+Ku
Interesting point. Should have mentioned this difference though, it's a quite different architecture with different tradeoffs.
◧◩◪◨
21. emmela+ED1[view] [source] [discussion] 2019-05-28 01:59:00
>>archgo+uJ
maildir is specified at https://cr.yp.to/proto/maildir.html and billions of message uses it each year. So that's pretty safe.

I can't vouch for the queueing code but I believe it's quite robust too.

◧◩◪◨⬒
22. kondro+Kl2[view] [source] [discussion] 2019-05-28 12:25:27
>>really+Ku
This is a message queue. But wouldn’t be fit for purpose for the way most people need to use message queues.
replies(1): >>really+IV2
◧◩◪◨⬒⬓
23. really+IV2[view] [source] [discussion] 2019-05-28 16:17:35
>>kondro+Kl2
What constitutes "the queue" is a matter of data structuring, not a singular brand (eg RabbitMQ, Redis, SQS, etc each have their own internals).

Any middle tiering of the data before it reaches the consumer, is still "the queue". You don't need to know the internals of SQS, anymore than a consumer need know the black box elements of how you collate the messages within your ad-hoc queue.

[go to top]