I think that demonstrates more about you than it does about him. Asking "how did you come to this number" is a valid question anytime someone gives you a number that would be hard to calculate. Asking for receipts is not the same as being a neo nazi..
I see extremists (on both sides) do this all the time, you don't argue the actual point you just say its "adjacent to bad thing, thusly bad"