The condition of being descendants of slaves, or people who faced other forms of official discrimination cited in the prevention of intergenerational wealth such as redlining, blockbusting or unfavorable treatment in the GI bill, etc., is ultimately an individual situation for each individual affected.
The idea that you can dismiss that as not an individual hardship -- though it kind of is for those impacted -- strikes me as pretty much a word game, nothing more. Not unlike the word games American laws started to use when they could no longer punish people de jure for their race.
I'm not staking out a position here, I haven't made up my mind myself. I'm just pointing out that OP raised a valid point which you didn't really address.