There could be functional redundancies or alternative systems at play that we haven't identified, systems that allow thought to access linguistic capabilities even when the specialized language areas are offline or unnecessary. The question of what "language in thought" looks like remains open, particularly in tasks requiring comprehension. This underscores the need for further exploration into how thought operates and what role, if any, latent or alternative linguistic functionalities play when conventional language regions aren't active.
In short, we may have a good understanding of language in isolation, but not necessarily in its broader role within the cognitive architecture that governs thought, comprehension, and meaning-making.
All other things being equal, its is a reason to provisionally reject the hypothesis that those kinds of thought use language as introducing entities (the ties between those kinds of thought and language) into the model of reality being generated that are not needed to explain any observed phenomenon.