Diary of a geek

May 2005
Mon Tue Wed Thu Fri Sat Sun

Andrew Pollock


Other people's blogs


RSS feed

Contact me

JavaScript required

Monday, 23 May 2005

World of pain (or 6th migration (fourth enforcement module))

Just when you thought it was safe to sleep past stupid o'clock...

I wrote last week that my firewall replacement program had been put on hold. Well, due to some internal politics, I ended up replacing the last remaining one in this particular gateway this morning, rather than leaving it with two arms on new firewalls and one arm on an old one.

I didn't actually mind too much, as it's kind of nice to actually vaguely finish something. I've only got two more in this site to do (in a separate gateway), but that'll happen probably in July at the rate things are going.

Anyway, I was in a world of hurt this morning performing this change, and it was all my fault.

I like to have a repeatable, auditable process for building these firewalls, and it's served me quite well to date. I think I need to more thoroughly check what I'm doing though. I did build this one in a bit of a rush because I only learned I was doing it last Wednesday afternoon, and did most of the configuration migration on Thursday.

The main source of problems was that I managed to migrate /etc/netmasks as /etc/defaultrouter. So not only did I bring up all my interfaces with completely insane netmasks, I also brought up my firewall with 8 default routes. This left me quite miffed, as I discovered after I'd swapped to the new firewall that I had completely bogus interface netmasks and a pre-migrated /etc/netmasks file. I couldn't figure this out, as I'd ticked off migrating /etc/netmasks on my checklist. It was only after a colleague came in and noticed the 8 default routes and that /etc/defaultrouter contained /etc/netmasks, that I realised what had happened.

I'm very annoying with myself for

  1. screwing up
  2. not realising pre-swap that I'd screwed up.

The moral of the story is that I need to actively recheck my checklist after I've completed it, not trust myself to have done each line-item correctly at the time.

At least there were no kangaroos this morning.

[17:07] [work] [permalink]