If you can retire at 40 having lived your 20s/30s to the fullest then game on, but it would be crazy to sacrifice that time when you are so free and full of energy otherwise IMHO.
FWIW I am fortunate enough to have really enjoyed by earlier years and be mostly retired in my early 40s.