The extremists on both sides are what you hear the most of, but the rest of the population is far more moderate.
Also I agree 100% with you that "one side" trying to pass laws to control access to books in libraries is exactly the same as the "other side" going around telling people not to buy tesla cars. Definitely not something to worry about.