zlacker

I recently needed to recommend some IAM permissions for an assistant on a hobby project; not complete access but just enough to do what was required. Was rusty with the console and didn't have direct access to it at the time, but figured it was a solid use case for LLMs since AWS is so ubiquitous and well-documented. I actually queried 4o, 3.7 Sonnet, and Gemini 2.5 for recommendations, stripped the list of duplicates, then passed the result to Gemini to vet and format as JSON. The result was perfectly formatted... and still contained a bunch of non-existent permissions. My first time being burned by a hallucination IRL, but just goes to show that even the latest models working in concert on a very well-defined problem space can screw up.

>>Jordan+(OP)
AWS docs have (had) an embedded AI model that would do this perfectly. I suppose it had better training data, and the actual spec as a RAG.

>>Jordan+(OP)
Listen I don't blame any mortal being for not grokking the AWS and Google docs. They are a twisting labyrinth of pointers to pointers some of them deprecated though recommended by Google itself.

>>dotanc+N4
Both AWS and Azure docs’ built in models have been absolutely useless.

>>Jordan+(OP)
Sounds like a vague requirement, so I'd just generally point you towards the AWS managed policies summary [0] instead. Particularly the PowerUserAccess policy sounds fitting here [1] if the description for it doesn't raise any immediate flags. Alternatively, you could browse through the job function oriented policies [2] they have and see if you find a better fit. Can just click it together instead of bothering with the JSON. Though it sounds like you're past this problem by now.

[0] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_poli...

[1] https://docs.aws.amazon.com/aws-managed-policy/latest/refere...

[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/access_poli...

>>Jordan+(OP)
by asking three different models and then keeping everything single unique thing they gave you, i believe you actually maximized your chances of running into hallucinations.

instead of ignoring the duplicates, when i query different models, i use the duplicates as a signal that something might be more accurate. i wonder what your results might have looked like if you only kept the duplicated permissions and went from there.