It's done recreationally, costs money, costs time, can cause injuries and joint problems, and is not productive. There are health benefits, but nothing that can't be had by much safer and less costly means of exercise.
So criticizing the concept of vice on the grounds that "everything that's fun is a vice" is somewhat of a semantic strawman - you're criticizing the word by changing its meaning.