Air is a particularly poor conductor of heat, your hands touching the product to move it around is doing more damage than anything else. Your fridge thermometer only measures air temperature, not product temperature, which is what you actually should care about.