World of pain (or 6th migration (fourth enforcement module))
Just when you thought it was safe to sleep past stupid o'clock...
I wrote last week that my firewall replacement program had been put on hold. Well, due to some internal politics, I ended up replacing the last remaining one in this particular gateway this morning, rather than leaving it with two arms on new firewalls and one arm on an old one.
I didn't actually mind too much, as it's kind of nice to actually vaguely finish something. I've only got two more in this site to do (in a separate gateway), but that'll happen probably in July at the rate things are going.
Anyway, I was in a world of hurt this morning performing this change, and it was all my fault.
I like to have a repeatable, auditable process for building these firewalls, and it's served me quite well to date. I think I need to more thoroughly check what I'm doing though. I did build this one in a bit of a rush because I only learned I was doing it last Wednesday afternoon, and did most of the configuration migration on Thursday.
The main source of problems was that I managed to migrate /etc/netmasks as /etc/defaultrouter. So not only did I bring up all my interfaces with completely insane netmasks, I also brought up my firewall with 8 default routes. This left me quite miffed, as I discovered after I'd swapped to the new firewall that I had completely bogus interface netmasks and a pre-migrated /etc/netmasks file. I couldn't figure this out, as I'd ticked off migrating /etc/netmasks on my checklist. It was only after a colleague came in and noticed the 8 default routes and that /etc/defaultrouter contained /etc/netmasks, that I realised what had happened.
I'm very annoying with myself for
- screwing up
- not realising pre-swap that I'd screwed up.
The moral of the story is that I need to actively recheck my checklist after I've completed it, not trust myself to have done each line-item correctly at the time.
At least there were no kangaroos this morning.