Just in semi-recent history we had mccarthy 'cancelling' people for purportedly being linked to communism, and that was a whole lot more serious than some modern publisher refusing to buy your book or twitter banning you.
A few decades before that, it wasn't real uncommon that if your neighbors objected to who you were or what you said, for them to hang you by your neck from a convenient tree until you were quite dead.
Humans have always suffered penalties for being on the wrong side of their neighbor's majority opinions. These days the penalties are frankly pretty minor.
It seems so strange to me what this politicized bubble has become. So far it's an (attempt at) collaborative "vote with your wallet" [0] and one political party is loudly saying "not like that". But the political party most complaining about "cancel culture" is also the party most actually trying to ban things, yet that's not "cancel culture" it is "think of the children" (and it's not "vote with your wallet", it is town hall grandstanding and letter writing campaigns and lobbyists).
It is such a fascinating example of hypocrisy in our society right now. To entirely strawman it: "You can't tell me what to do [with my cash], but I can tell the libraries what you shouldn't be allowed to read. You are the real monster telling me what to do with my cash. Censorship of libraries is in the best interests of the children! Think of the children! They could be reading filth, oh no! Freedom of speech doesn't apply to children, just to me!"
I know in many cases not everyone that hates "cancel culture" also wants to ban library books, but the intersection seems large enough that it is concerning.
[0] Which carries its own terrible baggage. "Vote with your wallet" just means that the rich "deserve" more votes. That's not Democracy. Which isn't to say that boycotts and general strikes don't work or don't have some power in our economy, but that it isn't always the power you think it is, and to wield that power correctly takes collective effort (large enough boycotts and general strikes to hit a bottom line figure), not individualism.
The extremists on both sides are what you hear the most of, but the rest of the population is far more moderate.
Also I agree 100% with you that "one side" trying to pass laws to control access to books in libraries is exactly the same as the "other side" going around telling people not to buy tesla cars. Definitely not something to worry about.