Mistakes were made.

Sometimes mistakes are made and bad things happen. In this story, all is well that ends well, but a company almost lost their data because of an oversight.

Enjoy

Client’s ▓▓▓▓▓▓▓ appliance’s warranty expires
Provider works with ▓▓▓▓▓▓▓ support to procure new appliance
New appliance shipped to client
Client unboxes new ▓▓▓▓▓▓▓ appliance, Provider goes to work configuring
New ▓▓▓▓▓▓▓ appliance configured, backups running successfully
Provider instructs Client to let backups run for 7 days, then ship back the old ▓▓▓▓▓▓▓ appliance
7 Days pass – successful backups
Client ships back old ▓▓▓▓▓▓▓ appliance for decommissioning
Provider receives old ▓▓▓▓▓▓▓ appliance and begins decom and wipe of old appliance
Alerts come in for failed backups, which the provider troubleshoots for 8 days
Provider begins further decommissioning of old appliance, stops at full system wipe until ▓▓▓▓▓▓▓ Sales updates the portal
Upon further investigation by the provider, the ▓▓▓▓▓▓▓ appliance is not on the same subnet as the server being backed up
Provider suspects the appliance got plugged into the wrong switch on the wrong subnet
Provider calls client, instructing them to move the ▓▓▓▓▓▓▓ to the correct switch
Provider left voicemail
Provider calls client again, informing them backups aren’t running until this is resolved
Client answers, informs Provider that they believe they have made a mistake
Provider, answering the client mid-sentence, “Yes, we need it moved to the other switch”
Client, answering: “No, I shipped you the new appliance.”
Provider:
Client:
Provider:
Client:
Provider: When did you discover this?
Client: When ▓▓▓▓▓▓▓ told me to go look at the box 2 days ago
Provider:
Provider:
Provider:
Provider: Alright sir, thanks very much for that information. I will get this appliance dispatched back out to you. Please ship the old one back.
Client: Sure thing, thanks.

TLDR; almost wiped entire org’s data, saved by uncharacteristic laziness
End