Many illegal drugs have a potential of causing harm.
Should the harm be purely physical?
Homelessness is a public nuisance, which we can choose to ignore, but so is walking naked in a school yard, which will be seen as a much more controversial subject.
But: - for it to work people should estimate the risks correctly - most people caught DUIing apparently are quite bad in understanding the risks involved
To the user, which is a risk they chose to take.
Now you could argue that the illicit trade is causing harm, but that's the direct result of criminalization and not of people choosing to consume substances.