Importantly the speaker and listener are not consciously aware of this happening. The net result is that you can say literal/plain thing A and the listener can hear literal/plain thing B.
Speaking to Americans requires a significantly accurate modeling of the listener's mind and expectations to be able to be clearly understood, much much moreso than any other language I have studied or even heard of.
Basically, it is very easy to be totally misunderstood when using plain, literal speech (such as is common in Germany or in Slavic countries).
I've written about it: https://sneak.berlin/20191201/american-communication/
You /can't/ communicate without euphemisms, and trying to will always fail and make you seem like a dick even though you're just being straightforward. That is likely where the difficulty you've experienced comes from.
(For context, your exact situation occurred this weekend. I was invited to an event and said yes, but both me and my friend knew that I would not attend)