So, today is my last day at Cybertrust. Much to my displeasure it started at 4am, when I had to come into work to perform a load balancer upgrade.
I'd previously joked with a few people that I was tempted to not bother with the upgrade and just say I'd rolled back. Funnily enough, that was exactly the outcome anyway...
Being an uber secure facility, we keep all our class C rack keys in an electronic key safe. This being a funky PIN access-restricted, battery backed, solenoid-driven thing.
So when I rolled up at work at silly o'clock, the first thing I went to do was go to the key safe to pull out the rack keys I'd be needing to access the relevant racks to perform this upgrade. They key safe's little LCD display was dead. I laughed. I then proceeded to get lots of people out of bed to try and track down the location of the spare set of keys that wasn't in the key safe.
So to cut a long story short, we burned all our troubleshooting time trying to get rack keys, and we had some strange problems with locally attached devices not ARPing correctly, so we had to roll back anyway.
So I might as well have rolled over and gone back to sleep at 4am when the alarm went off. At least I'll get to knock off work early.