zlacker

The privacy aspect and other security risks tho? So far all the praise I hear on productivity are from people using cloud-hosted models.

Claude, Gemini, Copilot and and ChatGPT are non-starters for privacy-minded folks.

So far, local experiements with agents have left me underwhelmed. Tried everything on ollama that can run on my dedicated Ryzen 8700G with 96GB DDR5. I'm ready to blow ~10-15k USD on a better rig if I see value in it but if I extrapolate current results I believe it'll be another CPU generation before I can expect positive productivity output from properly securely running local models when factoring in the setup and meta.

>>baobun+(OP)
This is probably the biggest danger. Everyone is assuming optimization work reduces cost faster than these companies burn through capital. I'm half inclined to assume optimization work will do it, but it's far from as obvious as they want to portray it.

>>baobun+(OP)
Privacy is not binary, and it would make it easier if you outlined specific scenarios.

Most providers promise not to train on inputs if used via an API (and otherwise have a retention timeline for other reasons).

I'm not sure the privacy concern is greater than using pretty much any cloud provider for anything. Storing your data on AWS: Privacy concern?

>>baobun+(OP)
Almost all of the cloud vendors have policies saying that they will not train on your input if you are a paying customer.

The single biggest productivity boost you can get in LLM world is believing them when they make those promises to you!

>>simonw+K7
> The single biggest productivity boost you can get in LLM world is believing them when they make those promises to you!

I'm having a hard time interpreting what you mean here. It sounds like something straight out of a cult.

>>JohnKe+bj
Or for someone trying to convince you to give your code to train on for free.

>>baobun+(OP)
MacStudio with 512GB RAM starts at around 10k and quantized DeepSeek R1 671B needs around 400GB RAM, making it usable for your needs. It produced some outstanding code on many tasks I tried (some not so outstanding as well).

>>storus+wl
Am I right in assuming that running Linux (or anything else than macOS) on the MacStudio is experimental at best?

I'd be looking for something that can run offline and receive system updates from an internal mirror on the airgapped network. Needing to tie an AppleID to the machine and allow it internet access for OS updates is a hard sell. Am I wrong in thinking that keeping an airgapped macOS installation up to date would additional infrastructure that requires some enterprise contract with Apple?

>>Beetle+W4
> Storing your data on AWS: Privacy concern?

Unencrypted? You bet.

>>baobun+Cn
IIRC you can download OS update/installation DMG from Apple, put it on a USB key and run it on airgapped system. I don't think you even need Apple ID. MacOS with homebrew works more-less like Linux, at least tooling is basically the same. You won't be able to install any Linux on M3 Ultra.

>>JohnKe+bj
An LLM vendor says to you "we promise not to train on your input". You have two options:

1. Believe them. Use their products and benefit from them.

2. Disbelieve them. Refuse to use their products. Miss out on benefiting from them.

I pick option 1. I think that's the better option to pick if you want to be able to benefit from what this technology can do for you.

Personally I think "these people are lying about everything" is a stronger indication of a cult mindset. Not everyone is your enemy.

>>simonw+XF
Well, I've been personally lied to about privacy claims by at least Google, Meta, Amazon, Microsoft. Some of which has been observed in courts. OpenAI communication has obviously been dishonest and shady at times if you keep track. All of the above have fallen in line with current administration and any future demands they may have to pin down or cut off anyone opposing certain military acts against civilians or otherwise deemed politically problematic. DeepSeek's public security snafu does not instil confidence that they can keep their platform secure even if they tried. And so on.

Fool me twice, you can't get fooled again.

>>baobun+(OP)
> So far, local experiements with agents have left me underwhelmed.

Devstral (mistral small fine-tuned for agentic use coding) w/ cline has been above expectations for me.

>>baobun+dK
The worst part to me is how little anyone seems to care about privacy - it just is how the world is. The US economy (or at least almost all e-marketing) seems to run on the idea that there's no such thing as privacy by default. Its not a subject that is talked about nearly enough. Everything is already known by uncle sam regardless. Its really strange, or maybe fortunate, that we're basically at a place that we often worried about but things haven't gone totally wrong yet. Corporate governance has been not that terrible (they know that its a golden goose they can't unkill). We'll see what happens in the next decade though - a company like google with so much data but losing marketshare might be tempted to be evil, or in todays parlance, have a feduciary responsibility to juice peoples data.

>>simonw+K7
> ...have policies saying that they will not train on your input if you are a paying customer.

Those policies are worth the paper they're printed on.

I also note that if you're a USian, you've almost certainly been required to surrender your right to air grievances in court and submit to mandatory binding arbitration for any conflict resolution that one would have used the courts for.

>>simonc+UT
How many paying customers do you think would stick around with an AI vendor who was caught training new models on private data from their paying customers, despite having signed contracts saying that they wouldn't do that?

I find this lack of trust quite baffling. Companies like money! They like having customers.

>>simonw+hY
If you pay attention, you see that the cost to large companies of reputational damage is very, very small. "The public" has short memories, companies tend to think only about the next quarter or two, PR flacks are often very convincing to Management, and -IME- it takes a lot of shit for an enterprise to move away from a big vendor.

And, those who are pay attention notice that the fines and penalties for big companies that screw the little guys are often next-to-nothing when compared with that big company's revenue. In other words, these punishments are often "cost of doing business" expenses, rather than actual deterrents.

So, yeah. Add into all that a healthy dose of "How would anyone but the customers with the deepest pockets ever get enough money to prove such a contract violation in court?", and you end up a profound lack of trust.

>>simonc+w01
Companies tend to be a lot more cautious at spending their money than consumers are.

This space is fiercely competitive. If OpenAI turn out to be training on private data in breach of contract, their customers can switch to Anthropic.

>>baobun+dK
on the other hand, if AWS or Microsoft was caught taking customer data out of their clouds their business would be over. I don't know if AI has anything to do with it, inference is just another app they sell.