That he was drunk makes it very likely that he was expressing what he really thinks. It's not a matter of "dragging him down" as much as "when someone shows you who they are, believe them."
Apologizing is great, but it can't make people unlearn something they learned about the person.