I've come to the conclusion that one needs to have a healthy distance to ones points, and realize that it's just a gamification of discourse that is helpful up to a certain point, but that it quickly gets counterproductive when that's mostly all that people care about.
I sense that many people are extremely strategic about what they submit or comment on, where they comment etc, and therefore get sensitive about the downvotes as well. Unfortunately it is a seemingly unavoidable consequence of introducing scores in any context.
Better just accept that people downvote you for whatever reason, don't worry too much about your karma (what on earth for?). Care about the content instead.