It's not 100%, but they'll get rid of like 80% of the background noise without playing anything. To get that last 20% yeah you need something playing to help mask it(even at an extremely low volume).
Sometimes I wear mine with nothing playing while at work just to get rid of random office chatter.
I do have noise canceling headphones and I like them, but they don't really do a lot for people talking.
My Bose QC35 set is ~4 years old, on their second set of muffs and only play sound when I fly, once or twice a year.