It's not 100%, but they'll get rid of like 80% of the background noise without playing anything. To get that last 20% yeah you need something playing to help mask it(even at an extremely low volume).
Sometimes I wear mine with nothing playing while at work just to get rid of random office chatter.