Also, if the 20 year old looks back 5 years, they were a hormonal teenager, still growing physically, treated as a minor and had never experienced half the things they'd experienced now. They were a completely different person in many respects then, and if they look back 15 years it's their very first memories.
If the 50 year old looks back 5 years, their life probably wasn't all that different. If they look back 15, their kids were still at home and they had a different job title but they were fundamentally the same person, and they feel they've been that way for a very long time.