Nah. Far more people use crypto for speculation than for actually illicit purposes.
Fiat money derives its value from what you say, indeed. It is traded on exchanges because of that. But fungibility is only one of several factors. Money should also serve as store of value, but how long is highly debatable and the point.