Uncoordinated changes; aka FUBAR

In the tech world, we have our fair share of keyboard cowboys and rogue technicians.

Over a socially distance drink yesterday, a buddy of mine told me about an incident he had where a technician decided to change the configuration of an existing VPN tunnel in their AWS instance. Without a service window. Without notification. Without planning.

Then, the expected happened – the tunnel went down and communications were lost.

My buddy received an urgent request: “VPN DOWN – NEED TO FIX”

So, as per tradition, he started looking at the tunnel. IKE mismatch. (This is a tunnel that has been up for years without issue). When he inquired what happened, he found out the technician saw that a tunnel was configured one way and hadn’t seen it configured that way before so he changed it. He thought, “the firewall would pick it up” when he made the changes.

Long story short, the changes were reverted after much chastising and the tunnel came back up.