failover

Last night/this morning I conducted a total brownout failover exercise.

What if everything in the world went down (or there was a major national grid outage, or similar, or Openreach forgot how to run a fibre network)?

What if someone dropped an aircraft on a regional power distribution centre?

What would happen to my extensive audio and video media library?

What would happen to my data?

Well, there’s only one way to find out (in a simulated kind of way, obv).

But first, before I did anything drastic, I plugged in my nice new shiny UPS unit.

And then I went to the MCB and flipped the switch.

And went to bed.

After six hours sleep I made tea and inspected the devastation and began reading the various server/device logs.

The router was dead, obv, but the NAS (in its capacity as DHCP router) was still up and showing full signs of alertness.

I could plug a monitor and keyboard directly in to HP Server A (I really need to name these things) and HP Server B and access the various media and data stores that are spread across these devices.

I could plug a laptop directly in to the NAS and access the operating system as an administrator. I could also access a partitioned area of the volume and and pick up any/all of the weekly contingency backup that was tucked away there.

And if I had integrated the router in to this setup, my WiFi and local area network would have been up and running for me to do all of these things from the comfort of my own bed.

But I deliberately left the router out of the protected environment, because it wouldn’t have been a real test, would it?

I took the quiet time as an opportunity to deploy a WordPress upgrade across the WP estate, and I triggered a non-standard backup on to my emergency external device.

Then I restored power.

The router came back up (logs showed no issues), and the local area network popped back in to place around the infrastructure.

And the UPS stood itself down and resumed charge mode.

So yeah, that was a pretty good test.

All server and system logs show no problems, and the router log merely records a hard shutdown.

Brilliant!

 

Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *