I think you hit the nail on the head - I think it's the stupid decision on Go lang designers part to make panic-s recover-able. This necessitates stack unwinding, meaning defer-s still need to run if a panic happens down the stack.
Since they didn't want to have a 'proper' RAII unwinding mechanism, this is the crappy compromise they came up with.