zlacker

>>mpweih+(OP)
One downside of SQS is that it doesn't support fan-out, for eg. S3->SQS->multiple consumers. The recommendation instead seems to be to first push to SNS, and then hookup SQS/other consumers to it. Kinesis/Kafka would appear to be better suited for this (since they support fan-out like SNS and are pull-based like SQS), but aren't as well supported as SNS/SQS (you can't push S3 events directly to Kinesis for eg.) Can someone from AWS comment on why that is? Also, related: when can we expect GA for Kafka (MSK)?

>>hexene+ci
Kinesis is not necessarily well-suited fan-out. It is very well suited for fan-in (single consumer, multiple producers).

Each shard allows at most 5 GetRecords operations per second. If you want to fan out to many consumers, you will reach those limits quickly and have to implement a significant latency/throughput tradeoff to make it work.

For API limits, see: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_...