Yes, that's exactly the problem with the concept of white fragility. It's a Kafka trap [0]. You either agree with the concept and its implications/assumptions or you are accused of exemplifying it. It's a clever bit of rhetoric if you're fan of argument by denunciation.
[0]: https://en.wiktionary.org/wiki/Kafkatrap