If I understand your "(recursively)" part correctly, this could make for a really interesting mass-shedding of users the first time anyone was banned. Like watching Filezilla delete a directory structure ...
I'm not certain how to deal with the problem of initial members / founders. Surely the founder would not set up the system so they would be banned the first time someone was banned. It seems there would need to be a set of unbannables, including the founder.
I suppose you would need to handle the detached leaves of the tree too. This brings the unbannables back into play. So if A was an unbannable and had invited B, B had invited C1 and C2 and C3, C1 had invited D and D got banned, C1 and B would also be banned, and C2 and C3 would be considered invited by A.
I think this is getting far too complicated to be practical.
But honestly, that leads to very complicated systems, lots of angry banned people, … anonymity, in moderation, is good.