Importantly the speaker and listener are not consciously aware of this happening. The net result is that you can say literal/plain thing A and the listener can hear literal/plain thing B.
Speaking to Americans requires a significantly accurate modeling of the listener's mind and expectations to be able to be clearly understood, much much moreso than any other language I have studied or even heard of.
Basically, it is very easy to be totally misunderstood when using plain, literal speech (such as is common in Germany or in Slavic countries).
I've written about it: https://sneak.berlin/20191201/american-communication/
You /can't/ communicate without euphemisms, and trying to will always fail and make you seem like a dick even though you're just being straightforward. That is likely where the difficulty you've experienced comes from.
(For context, your exact situation occurred this weekend. I was invited to an event and said yes, but both me and my friend knew that I would not attend)
This reminded me of this infamous bit from Yes, Minister, and although it's not actually entirely an example of this, it's too good not to share now i've found it:
Sir Frederick: There are four words to be included in a proposal if you want it thrown out.
Sir Humphrey: Complicated. Lengthy. Expensive. Controversial. And if you want to be really sure that the Minister doesn't accept it, you must say the decision is "courageous".
Bernard: And that's worse than "controversial"?
Sir Humphrey: Oh, yes! "Controversial" only means "this will lose you votes". "Courageous" means "this will lose you the election"!
There is a reason america is number 1 in confidence, but ranked 25th in math and 21st in science out of the top 30 developed countries.