Assuming that refusal-of-service is a sort of relief valve is wildly optimistic.
Overt but relatively passive forms of racism effectively give pervasive comfort and encouragement to those who would engage in more active acts. Indirectly, this is also why "dog whistle" speech is so dangerous.
Per your example, in an environment where simply refusing service to you was common and widespread, you might find that someone who does agree to serve you is doing so just for the opportunity to spit in your food (at best).