Is this not how it works? I'm not Muslim, this is what I understood from a Muslim coworker. He also implied that there's some politics involved, in that various Middle Eastern countries align themselves with their political bloc when announcing the day, although any political motivations are "alleged" and not acknowledged.
However there are legal schools and councils that people trust. Legal schools (or fiqh) were established by famous scholars that nearly everyone recognizes as an expert and were carried on by their students.
From a political standpoint, a lot of Muslims defer to Saudi Arabia because they are the current caretakers of Mecca and Medina.
But generally speaking for everyday normal Muslims they go with whatever their local imam says and go with whatever method the imam used because they are usually the most knowledgeable person around on such matters.