zlacker

I've worked almost exclusively on a large Golang project for over 5 years now and this definitely resonates with me. One component of that project is required to use as little memory as possible, and so much of my life has been spent hitting rough edges with Go on that front. We've hit so many issues where the garbage collector just doesn't clean things up quickly enough, or we get issues with heap fragmentation (because Go, in its infinite wisdom, decided not to have a compacting garbage collector) that we've had to try and avoid allocations entirely. Oh, and when we do have those issues, it's extremely difficult to debug. You can take heap profiles, but those only tell you about the live objects in the heap. They don't tell you about all of the garbage and all of the fragmentation. So diagnosing the issue becomes a matter of reading the tea leaves. For example, the heap profile says function X only allocated 1KB of memory, but it's called in a hot loop, so there's probably 20MB of garbage that this thing has generated that's invisible on the profile.

We pre-allocate a bunch of static buffers and re-use them. But that leads to a ton of ownership issues, like the append footgun mentioned in the article. We've even had to re-implement portions of the standard library because they allocate. And I get that we have a non-standard use case, and most programmers don't need to be this anal about memory usage. But we do, and it would be really nice to not feel like we're fighting the language.

>>openas+(OP)
I've found that when you need this it's easier to move stuff offheap, although obviously that's not entirely trivial in a GC language, and it certainly creates a lot of rough edges. If you find yourself writing what's essentially, e.g. C++ or Rust in Go, then you probably should just rewrite that part in the respective language when you can :)

>>openas+(OP)
I guess you'd be interested in the arena experiment, though it seems to be currently on pause

>>openas+(OP)
Perhaps the new "Green Tea" GC will help? It's described as "a parallel marking algorithm that, if not memory-centric, is at least memory-aware, in that it endeavors to process objects close to one another together."

https://github.com/golang/go/issues/73581

>>theobe+88
I saw that! I’m definitely interested in trying it out to see if it helps for our use case. Of course, at this point we’ve reduced allocations so much the GC doesn’t have a ton of work to do, unless we slip up somewhere (which has happened). I’ll probably have to intentionally add some allocations in a hot path as a stress test.

What I would absolutely love is a compacting garbage collector, but my understanding is Go can’t add that without breaking backwards compatibility, and so likely will never do that.

>>openas+(OP)
I know this comment isn't terribly helpful, so I'm sorry, but it also sounds like Go is entirely the wrong language for this use case and you and your team were forced to use it for some corporate reason, like, the company only uses a subset of widely used programming languages in production.

I've heard the term "beaten path" used for these languages, or languages that an organization chooses to use and forbids the use of others.

>>openas+(OP)
Embed a local redis or sqlite instance?

>>daxfoh+Jv
Using SQLite in Go is also not for the faint of heart either.

>>andrew+Ao
No, Go isn’t actually that widely used at my company. The original developers chose Go because they thought it was a good fit for our use case. We were particularly looking for a compiled language that produces binaries with minimal dependencies, didn’t have manual memory management, and was relatively mature (I think Rust was barely 1.0 at the time). We knew we wanted to limit memory usage, but it was more of a “nice to have” than anything else. And Go worked pretty well. It was in production for a couple years before we started getting burnt by these issues. We are looking at porting this to Rust, but that’s a big lift. This is a 50K+ line code base that’s pretty battle tested.

>>openas+4T
> The original developers chose Go because they thought it was a good fit for our use case.

I don't completely get this. If you are memory requirements are strict, this makes little to no sense to me. I was programming J2ME games 20 years ago for Nokia devices. We were trying to fit games into 50-128kb RAM and all of this with Java of all the languages. No sane Java developer would have looked at that code without fainting - no dynamic allocations, everything was static, byte and char were the most common data types used. Images in the games were shaved, no headers, nothing. You really have to think it through if you got memory constraints on your target device.

>>openas+(OP)
> One component of that project is required to use as little memory as possible, and so much of my life has been spent hitting rough edges with Go on that front.

You made a poor choice of language for the problem. It'd be a good fit for C/C++/Rust/Zig.

>>p2deta+9Z
Wow. I bet that required some decent clever thinking that you would have only typically thought about in C or C++.

>>p2deta+9Z
The strictness of our memory requirements wasn’t made apparent until years later. For background, the application is a system daemon running on end-user servers. So every byte we allocate, every cycle of CPU we use, is a byte or cycle taken away from our customers. We don’t provide firm guarantees or formal SLAs on our memory usage, but we do try to minimize it whenever possible. Because what we don’t want is for someone to upgrade agent versions and suddenly our daemon is using significantly more memory and causing OOMs, or using more CPU and force customers to scale out their fleet. Our p99 cpu and memory usage right now are more or less the same as they were two years ago (RSS is under 40MB last I checked)

So it’s not like we’re running on a machine with only kilobytes of RAM, but we do want to minimize our usage.

>>andrew+Ao
The legendary “it is good enough for google, why shouldn’t we use it?”

>>openas+4T
Rust is bad at avoiding memory fragmentation in my experience.

If you use vanilla rust with global alloc everywhere then you will get fragmentation

You have to use nightly for custom allocators and implementing allocators and using them is super annoying still.