It's a different sort of 'passive'. There is this thing called
https://en.wikipedia.org/wiki/Default_mode_network which 'lights up' in very different ways, depending on what you do, and how. One could argue that is to be expected, because different regions of the brain are exercised. But that's not all of it. Regarding exercise, think of 'Use it, or lose it'. It's mostly the imagination which
is exercised while reading. If it isn't, it shrivels.