Maybe enable a pre-charged capacitor to the LED whenever the circuit is activated? A "minimum duty cycle" for the LED might help solve this.