AWS does not offer great discounts on the GPUs at this point, as they don't have nearly enough of them to meet demand. I'm no longer at a startup, and have worked at a couple of larger companies.
That said I'm mostly responding to the "two guys in a garage" comment with this. Larger companies are going to have different needs altogether.