I think it would be interesting to have to say why you up or down voted something, along with the vote. Make the reasons public, so people can see how the system is being used, and then publish guidelines based on the aggregate results that give people an indication of the best way to use the system.
It would probably not be ideal to publicly publish names along with reasons, since this might encourage flame wars about why people voted in certain ways. However, perhaps there could be more private means of dealing with people who consistently misuse/abuse the system.