That's too bad, because you can.
Stop worrying about your karma.
Naturally the reverse is also true and you can't help but not want either number to fall. In my case at least these feelings are involuntary, despite the fact that I know these are pretty much worthless numbers. (Except for the fact that karma average does effect the weight of your votes.)