What about alcohol use then? Smoking? Buying high risk stocks, options and NFTs? Investing in high-risk startups? Working for a high-risk startups? Spending 100 hours per week on work and neglecting one's family? Any of these could potentially lead to very sad consequences for not only the individual involved but for the people close to them. But once you step on this road, it can lead you to a very weird places if you're not careful. Or you may throw the consistency out of the window and just say "but this is different!" - but then I'd welcome you to explain how exactly it's different.
> but it quickly devolves from there.
This is a so called "gateway" theory, and there are many indications it is false. For example: https://www.rand.org/pubs/research_briefs/RB6010.html
I suspect that with further legalization and de-stigmatization of marijuana use, the link would become even weaker, because most users won't devolve anywhere - as most people who drink a can of beer on a weekend do not become raging alcoholics - and the cases where a person is driven to drug use by some problems not produced (though also not solved but frequently worsened) by drugs would be recognized as such instead of blaming the evil weed for everything. (NB: not a user myself, never did, never planning to)