Imprisonment is meant for rehabilitation in addition to punishment. There’s the idea, at least in theory, that people who commit crimes can eventually be functional members of society with full rights given a second chance. So we send people to prison and then let them resume their lives as citizens afterwards. If they owe money due to a civil suit they can still vote because why wouldn’t they? Franchise isn’t tied to financial means and shouldn’t be.
Don't forget the third big part: stopping them from violating the rights of others.
They do temporarily lose some rights, they do (and should) get them back when their "debt to society" is paid (which I find a slightly weird term, but whatever), why shouldn't the right to vote be one of the rights that you get back when you're rehabilitated and reintroduced into society, just like your right to freely move about?