zlacker

I think the distinction between programmatic error (solvable) and operational characteristic (mitigatable) is valid in theory, but I disagree that it matters in practice.

Proactive prevention (like bounds checking) only "solves" the class of problem if you assume 100% developer compliance. History shows we don't get that. So while the root cause differs (math vs. probabilistic model), the failure mode is identical: we are deploying systems where the default state is unsafe.

In that sense, it is an apples-to-apples comparison of risk. Relying on perfect discipline to secure C memory is functionally as dangerous as relying on prompt engineering to secure an LLM.

>>jfyi+(OP)
Agree to disagree. I think that the nature of a given instance of a programmatic error as something that, once fixed, means it stays fixed is significant.

I also think that if we’re assessing the likelihood of the entire SDLC producing an error (including programmers, choice of language, tests/linters/sanitizers, discipline, deadlines, and so on) and comparing that to the behavior of a running LLM, we’re both making a category error and also zooming out too far to discover useful insights as to how to make things better.

But I think we’re both clear on those positions and it’s OK if we don’t agree. FWIW I do strongly agree that

> Relying on perfect discipline to secure C memory is functionally as dangerous as relying on prompt engineering to secure an LLM.

…just for different reasons that suggest qualitatively different solutions.