zlacker

>>nico+(OP)
Our solve is to allow it to work with a local dev database and it's output is a script. Then that script gets checked into version control (auditable and reviewed). Then that script can be run against production. Slower iteration but worth the tradeoff for us.

Giving LLM even read access to PII is a big "no" in my book.

On PII, if you need LLMs to work on production extracted data then https://github.com/microsoft/presidio is a pretty good tool to redact PII. Still needs a bit of an audit but as a first pass does a terrific job.

>>fhub+gz
The script method is great, and it's generalisable to things outside of DB access.

E.g. I used this method when I wanted to carry out a large (almost every source file) refactoring of Cytoscape.js. I fed the LLM a bunch of examples, and I told it to write a script to carry out the refactoring (largely using regex). I reviewed the script, ran the script, and then the code base was refactored.

At the time, agents were not capable enough of doing large-scale refactors directly, as far as I was aware. And the script was probably much faster, anyway.