So let's say you have a generally fair time bonus for mild (clinical) anxiety. The issue is that it's fair for the average mild anxiety, it's an advantage if a student has extremely mild anxiety.
As you say, hopefully the test is not overly time focused, but it's still an advantage, and a lot of these students / parents will go for every advantage they can.
If thinking speed is not important, why are we evaluating it at all?
We might as well make races longer for athletes with longer legs. It’s unfair to the ones with shorter legs to have to move them more often.
We look at the range of lengths that is typical for legs. And all these get to compete under typical conditions.
Now let's say someone has a leg length that is fairly outside of the typical range. Let's say someone has a leg length of zero. We let these athletes compete with each other as well with different conditions, but we don't really compare the results from the typical to the atypical group.