Shouldn't be if your building was designed properly, with a proper Heat Recovery Ventilator (HRV) e.g. [0]. The long and short of it, fresh air is slightly heated by stale outgoing moist air.
[0] https://www.nrcan.gc.ca/energy-efficiency/products/product-i...