One possible way to address this is to make visible a list of users who flagged a post. The arguments against this are obvious. But without such information, in the end you have to accept one result of anonymous moderation is the generation of conspiracy theories.
It's a tradeoff, of course.