Is it within your rights ? totally. Does it make sense from a business perspective ? yes, probably. Is it morally right ? I'd say no. Will most people give you a damn about it ? probably not.
Most people won't care if you discriminate against some minority they're not part of and don't interact with. Some will, but I'm not sure how much it matters to you if you're seen as a "bad person" either way ?