I think it's only a real issue when you're coming from a language that has different rules. Block-scoping (and thus not being able to e.g. conditionally remove a temp file at the end of a function) would be equally surprising for someone coming from Go.
But I do definitely agree that the dynamic nature of defer and it not being block-scoped is probably not the best