I'm not sure I understand what you mean. Regardless of whether you "were not directly harmed" I don't see why someone should or shouldn't “say bad things about them."
Why shouldn't I express my opinion? Or are we in "If you don't have anything nice to say, don't say anything at all" territory?
I may be misunderstanding your point. If so, please do correct me. If not, I don't see why I (or anyone else) shouldn't express their opinion WRT anything.
What value that opinion may have can certainly be debated, but why should someone not express their opinion?