zlacker

Companies that ingest logs generally rip their customers faces off with their pricing. At least oTel spans can be tail-sampled.

>>tptace+(OP)
I worked out that it's cheaper to write logs to high-end Samsung SSDs and then throw them away every month than to retain them in the log analytics systems of some cloud services for the same period of time.

Wait, no, sorry... that doesn't quite "paint the right picture".

The "single use" SSDs are 75 times cheaper than storing the data in the cloud.

>>jiggaw+wC
As someone who works at a platform company that operates several (very) large log ingestion systems, if you're not indexing the logs usefully, having stored them on SSDs isn't doing much for you. It's just a weird comparison to make, is all I'm saying.

>>tptace+P02
Indexing that is 75x the original uncompressed data volume?

Because then I might accept the cost!

Realistically all of these systems use some type of data compression such as Parquet files, so the data on disk is likely smaller than the ingested data.

>>jiggaw+0x2
My point is that you're not really just paying for "storage".

>>tptace+6E2
What am I paying for?

I worked out that the markup on CPU, network bandwidth, and storage for the default logging products from the major clouds is on the order of 25x to 500x.

Okay, sure, there's some people that need to be paid, the back-end software may have some licensed components, etc, etc...

But still, comparing this to any other cloud service, the gross profit margin is just ridiculous!

It's the typical IT marketing trick of selling the commodity (VMs) at competitive prices, and then clawing back the profits via the "enterprise add-ons".

>>jiggaw+Z43
I mean: we faced the same buy/build issue, and we went "build", and while I wouldn't say it was our worst decision, it was not one of our best either. Certainly the problem doesn't simply boil down to NVME gigabyte costs!

That said: we both agree, log ingestion services are extremely expensive.