zlacker

I used to do remote work in firewalls quite often, and after locking myself out once or twice, I came up with a new habit: before making any changes I would schedule a reboot for 5min out which would revert any changes. That way if I locked myself out I could just wait for the reboot and get back in.

>>Nifty3+(OP)
And then if it worked for those 5 min before the reboot, you'd redeploy the change 'for real', without a reboot?

>>Nifty3+(OP)
This is clever, I like it.

>>dang+9b
Yeah, there are different kinds of memory in firewalls. Like a running-config and a startup-config. If you just change the running-config and don't commit to the startup-config, when the reboot takes place it'll pull the config from the (non-modified) startup-config instead, reverting changes.

>>Nifty3+(OP)
Standard practice on Cisco routers, where I've worked, is to do "reload 5" before doing dangerous things.

On juniper, it's "commit confirmed".

>>prox+Jb
"commit confirmed" from Juniper routers is much better

>>Nifty3+(OP)
‘sleep 300 && init 6’ was my go-to, but since then systemd has made firing init 6 unreliable (it won’t trigger a reboot locally if root has an open ssh session, at least on Ubuntu).

>>dang+9b
My typical workflow was:

- Schedule the reboot

- do my changes

- Make sure everything was working properly

- Go get lunch

- Notice a bunch of pages and alarms about a firewall going offline

- Rush back to my office

- Login to the firewall

- Schedule the reboot

- Re apply the changes

- Test it again

- CANCEL THE FING REBOOT THIS TIME

- Eat my now cold lunch

>>2snake+Ce
copy run start!

>>knorke+Jh
or safe mode on mikrotik

>>knorke+4i
Mikrotik safe mode gets a 3/5 in comparison - it reverts changes you made if you lose connection to the router, so it does it’s job as an anti-lockout mechanism; but I much prefer the atomic nature of a confit commit on junos still.

>>Nifty3+(OP)
I did a similar thing in the early days of my career, but I actually caused an outage as a result.

In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW.

The script runs, everything looks good. Five minutes later, pagers start going off.

Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.

>>Nifty3+(OP)
Kinda like changing display settings in Windows.

Changes will revert in 15 seconds....

>>Nifty3+kx
This used to be my workflow as well. I did make a few improvements though

  - begin change control at 4:55pm on Friday before Christmas
  - schedule reboot
  - paste changes
  - make sure everything is working properly
  - leave security key on desk 
  - go to christmas party
  - firewall goes offline, pages go off
  - remotely log into firewall with phone
  - rush back to office to get security key
  - accidentally type init 1 hanging server
  - discount datacenter remote hands not picking up the  phone
  - rush to datacenter to power cycle server
  - :(