zlacker

>>ustad+(OP)
I've worked almost exclusively on a large Golang project for over 5 years now and this definitely resonates with me. One component of that project is required to use as little memory as possible, and so much of my life has been spent hitting rough edges with Go on that front. We've hit so many issues where the garbage collector just doesn't clean things up quickly enough, or we get issues with heap fragmentation (because Go, in its infinite wisdom, decided not to have a compacting garbage collector) that we've had to try and avoid allocations entirely. Oh, and when we do have those issues, it's extremely difficult to debug. You can take heap profiles, but those only tell you about the live objects in the heap. They don't tell you about all of the garbage and all of the fragmentation. So diagnosing the issue becomes a matter of reading the tea leaves. For example, the heap profile says function X only allocated 1KB of memory, but it's called in a hot loop, so there's probably 20MB of garbage that this thing has generated that's invisible on the profile.

We pre-allocate a bunch of static buffers and re-use them. But that leads to a ton of ownership issues, like the append footgun mentioned in the article. We've even had to re-implement portions of the standard library because they allocate. And I get that we have a non-standard use case, and most programmers don't need to be this anal about memory usage. But we do, and it would be really nice to not feel like we're fighting the language.

>>openas+TK
I know this comment isn't terribly helpful, so I'm sorry, but it also sounds like Go is entirely the wrong language for this use case and you and your team were forced to use it for some corporate reason, like, the company only uses a subset of widely used programming languages in production.

I've heard the term "beaten path" used for these languages, or languages that an organization chooses to use and forbids the use of others.

>>andrew+t91
No, Go isn’t actually that widely used at my company. The original developers chose Go because they thought it was a good fit for our use case. We were particularly looking for a compiled language that produces binaries with minimal dependencies, didn’t have manual memory management, and was relatively mature (I think Rust was barely 1.0 at the time). We knew we wanted to limit memory usage, but it was more of a “nice to have” than anything else. And Go worked pretty well. It was in production for a couple years before we started getting burnt by these issues. We are looking at porting this to Rust, but that’s a big lift. This is a 50K+ line code base that’s pretty battle tested.

>>openas+XD1
Rust is bad at avoiding memory fragmentation in my experience.

If you use vanilla rust with global alloc everywhere then you will get fragmentation

You have to use nightly for custom allocators and implementing allocators and using them is super annoying still.