Like every other industry, there's a growth period where things are new and prices are reasonable, and then there's the "squeeze" where bean counters come in, make charts that are likely bs, and explain how much easier it'd be if we charged 4x as much for half the customer base.
Twitter was one of the first to give access to cheap mass data, and now they're one of the first to charge through the nose for that. The move is going to be that if you're not enterprise level you're not getting this data anymore, and I doubt it stops with reddit.