To create an analogy, my car doesn't have bullet proof glass, someone could easily shoot it up and i'd be dead. But nobody really goes around shooting up cars, so is it an issue?
Does this mean we should stop worrying about hardware bugs? I don't know the answer to this question. A principal engineer in the group that does Intel's hardware security validating and pentesting told me that they felt their job was to maintain the status quo of hardware bugs being harder to exploit than software bugs. More security than this is probably not justified from a risk vs cost analysis perspective; while less security than will probably break a lot of assumptions that people designing software make.