Think about the RLHF component that trains LLMs. It's the training itself that generalises - not the final model that becomes a static component.