Indeed, allowing this to occur has wrought orders of magnitude more death and destruction than sports gambling or drug use or prostitution.
no victim == no crime
I would very much like to believe that. But see what happened in Oregon after decriminalizing drugs.
They are not the same thing. One causes huge amounts of murder and violence, and the other is simply people destroying their own selves, as is their right.
Almost all of the gun crime in the US is the direct result of the prohibitions on the sale and manufacture of drugs.