As you say though, both are plausible, and that ambiguity is part of what makes the phrase so interesting.