if your entire family is "interrogated" over some crime you did not do, is that "punishment"? Maybe not legally, but it's a stressful time that may or may not cause strains on your relationship between your spouse and kids. And it's not like this is an investigator coming in for tea and asking a few questions, either.
morally, it would indeed be a punishment. There is now a bunch of nervous sentiment among your family for no reason, all because you were at the wrong place at the wrong time. Or if we want to be frank, you looked the wrong race at the wrong place at the wrong time.