If that is the case, why exactly don't researchers stick their papers literally anywhere on the internet that isn't a paywall? Is there a legal difference between sharing a paper with an individual and uploading it to a public index?
Honestly, with the current psychology of the public, I don't think this would be worse than having journals pick holding research for ransom.
Yes but in most cases, the authors are only legally allowed to give away the less desirable pre-prints manuscripts that was created for journal submission. The publishers allow them to retain the copyright to their draft work. However, the copyright to the final journal article with all the nice typesetting, pretty graphs, etc is assigned to the publisher. That more desirable finished article is the one behind the paywall.
(That's not to say that some authors "break the rules" and share the final journal article instead of their preprint but that's going to depend on the particular author.)