Exactly right, and in fact I am often excluded from whiteness on account of being Jewish. The most explicit white supremacists construct Whiteness very narrowly. And it's in opposition to white supremacy that a notion of Blackness is more important: not how they see themselves, but how they are treated, and how that treatment affects their lives.
I would be considered white in most circumstances so I get a pass from a lot of discrimination that affects others. And that's why I consider it important to take a stand against that discrimination -- though unfortunately, the people propagating the worst of that discrimination see me as a traitor.