Any time vouching triggers extra attention, the decision is recorded in the database. If someone routinely vouches and gets overruled (i.e. vouching for bad content), then their vouching no longer counts in the future.
At some point after the system is introduced, start giving extra weight to people whose vouching decisions line up with moderators.
Worst case, this is just flagging stuff for extra moderation attention so there's not a lot to abuse. If it's requiring too much extra attention, adjust the required vouch:flagged ratio or raise the threshold needed to vouch "For Real".
(I'm not saying I see anything wrong with the current system - I tend to appreciate how well this particular Walled Garden is tended to. But the vouch idea seemed cool to me, and I felt like I could contribute a useful implementation)