I've had my share of suspicions of "downvote brigades" and the like but from the point of view of an ordinary user they are indistinguishable from uncoordinated users expressing their disapproval so I guess it is best to just assume good faith.
It is actually fascinating to know how much of this type of behaviour is visible to the moderators and how big of a problem it is (and of course they too can't see the full extent of it).