In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW.
The script runs, everything looks good. Five minutes later, pagers start going off.
Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.