Society is not a game or stadium. There is no outside.
Justice is imperfect.
Laws are not all as obvious as 'breaking the bat'.
Now, responding to the part of your comment that isn't the shitty, specious analogy. You beg the question, saying that felons don't get to vote because they've opted out of civilized intercourse. You don't bother to argue the antecedent, you just assume it. That doesn't address the question being asked in the thread, it just affirms the way things are.