This is categorically untrue – if some past employer told you that, you might want to ask what their motives for lying to you were. Your knowledge of who you worked for is not corporate IP.
The actual legal standards vary from state to state but in some states it come down to three things: does that list have economic value on its own, would it be hard to recreate, and does the company make an effort to keep it secret? That probably won’t apply to your personal memory of who you worked for since that's highly unlikely to be an independently valuable resource - typically that would be a big list of non-public information like people who signed up to preorder a product, people with a certain need or interest, etc. – and it definitely wouldn’t include anything listed on their website, press releases, or someone’s C.V. If you dump the CRM on the way out, yes, you might be in trouble but there’s no legal standard expecting you to be mind-wiped on the way out.