I agree that there needs to be better control over such things. Whatever comes from this, it also needs to avoid the fundamental problem with robots.txt: it's advisory only, and bots can freely ignore it. And many do.
A real solution has to have an effectiveness greater than just asking nicely and hoping that people are honorable.