But a much easier argument against sports betting is that it ruins the sports. Players throw. They get good at subtly cheating. The gambling apparatus latches itself to the sport, to the teams and players, the umpires and judges, the sporting organizations. With this much money on the line, it's not a matter of if but when games are thrown, cheated -- the bigger the game, the bigger the incentive. It's even easier now because of the amount of side/parlay betting that is available. It exhausts the spirit of competition.
Sports gambling is diametrically opposed to sport itself.
There's a false dichotomy between prohibition and laissez-faire, which the US seems particularly prone to. You've seen similar issues with the decriminalisation of cannabis, where many states seem to have switched abruptly from criminalisation to a fully-fledged commercial market. There is a broad spectrum of other options in between those points that tend to be under-discussed.
You can ban gambling advertising, as Italy did in 2019. You can set limits on maximum stakes or impose regulations to make gambling products less attractive to new customers and less risky for problem gamblers. You can have a single state-controlled parimutuel operator. Gambling does cause harm - whether it's legal or not - but it is within the purview of legislators to create a gambling market in which harm reduction is the main priority.
Gambling, however has previously in the U.S. shown to be the leading cause of suicide attempts (20% in total) among all forms of addiction [1]. A body of evidence has also demonstrated it leads to divorce, bankruptcy, poor health and sometimes incarceration. Worth noting many of these studies centered around machine gambling and all forms of gambling are unique in terms of tendency for compulsion. Considering the landscape it is quite difficult for me to see a way of regulating out of this, not in the U.S. at least.
[1] Zangeneh and Hason 2006, 191-93
If you read some papers on the subject it should be plenty apparent that it has adverse effects on the development of young adults, as well as long term use by anyone, particularly of recent high-potency strains.
It's not as bad as other drugs (heroine), and it's worse than others (coffee), but it's not harmless. I'm far from being a prohibitionist, and live somewhere that has (I think) sensible policies (The Netherlands), but to simply put that it's "fairly harmless" as something most physicians agree with is not true. I'd say it's similar to alcohol in terms of its moderate use being possible in a working society - albeit with some negative outcomes for people that overdo it, or do it too early in life.
Edit: there's lots of discussion below about if the studies that exist are trustworthy or not, but since anyone can google for studies, I'll leave a different recommendation to check out the r/Leaves subreddit, and read some first hand accounts of long term and heavy users. It's at least a different type of source and you can make up your own mind about what real users say about it, in case you never encountered it before.