Hard to predict. The resulting chaos could cause outsized environmental damage. On the other hand, the environmental damage caused by wars tends to be short-term (in contrast to the damage caused by, say, fossil-fuel-based energy production and agriculture, which is perpetually-increasing damage to feed the needs of a society with a stable population and status quo).
But you're making an excellent point. I'm certainly not advocating for societal collapse. I, for one, think the risk of such is a bit overblown (societies tend to adapt, not implode, even in the face of demographic turbulence).
Besides, automation's supposed to make up for much of the labor shortfalls anyway.