He's talking about average rather than median precisely because his point isn't that "the economy grew, but the growth was captured by the wealthy," but rather "the economy hasn't grown much per capita." So you want to look at average, not median.
And he isn't saying that trends didn't continue, he's saying trends went one direction, but the policy discussion is as though the trends went in exactly the opposite direction, and that that's important.