Diary of a geek

September 2023
Mon Tue Wed Thu Fri Sat Sun
25 26 27 28 29 30  

Andrew Pollock


Other people's blogs


RSS feed

Contact me

JavaScript required

Friday, 06 March 2015

Everything old is new again

In 2005, when I was working for Cybertrust, my job title was Security Engineer. Then I left that job, and Australia, to go work for Google as a Linux Systems Administrator.

10 years later, after a fabulous 13 month break, I return to Google as a Security Engineer once more.

I honestly didn't expect it would be possible to return to Google under the same conditions that I had before I left, so I had very low expectations. I have to say that I am extremely humbled by the level of support I have received, both from my peers and my management, to come back.

I also had to make a difficult decision (aided by a coin toss) over whether to return to Google or accept an offer from a local company that would have been a good opportunity to make a broad impact.

[22:48] [work] [permalink]

Sunday, 18 January 2009

A small mention in the press

The project that I'm responsible for got a very tiny mention in this bio of Mark Shuttleworth in the New York Times.

[10:34] [work] [permalink]

Wednesday, 12 July 2006


[21:42] [work] [permalink]

Friday, 30 June 2006

It's not every day a billionaire blades by

So yesterday I was walking over to the main campus, (to avail myself of the on-site dentist of all things), and I noticed someone gliding down the road that the road I was walking along intersected with. Not seeing his feet, or any evidence of handlebars, I was vaguely interested in the mode of transport this (presumed Googler) was using.

As he got closer to the T-junction, I discovered he was on rollerblades. This was cool, and reminded me that I needed to try blading into work myself soon.

Aforementioned person then mounted the footpath that I was walking along, and as he passed me, I realised it was none other than Sergey Brin himself, out and about. Very cool.

So that's my latest brush with fame.

[21:37] [work] [permalink]

Tuesday, 27 June 2006

WTF mate?

This is just plain weird.

[16:38] [work] [permalink]

Saturday, 24 June 2006

Google Engineering Offsite 2006

So I went to my very first (and apparently the last ever in this format) Engineering Offsite on Thursday.

I somewhat foolishly, chose to cycle to it, rather than taking a bus, but that probably helped make it more memorable.

I was fairly stuffed afterwards, and so I didn't do a hell of a lot once I got there. I went on three of the rides back to back (the steel rollercoaster, the crusty old rickety wooden rollercoaster, and the Fireball, a nasty spinning, swing you side-to-side thing). In hindsight, that wasn't such a brilliant idea, as I spent the bus ride home feeling rather bilious indeed.

[00:49] [work] [permalink]

Wednesday, 12 April 2006

Learning Python

So I now work in Python shop.

I'm a Perl weenie from way back, and I've been meaning to learn Python for a while now, mainly because all the cool people were writing all the cool back-end stuff in Debian in it, and so I wasn't cool just writing Perl and PHP (just had to get that third P under my belt).

So I bought Learning Python a couple of years ago, and got through maybe a quarter of it before I got distracted with Uni or something and put it down again.

Now a large part of my job involves the care and feeding of a behemoth in-house developed system written in Python, so I have had to bite the bullet a little more and just learn it. The entire time, I've tried my level best to maintain an open mind about Python.

I'm starting to realise that the code-base I'm working on isn't necessarily the best introduction to Python. It's been through two iterations, and some of the first version has been pretty much cut and pasted into the second version, and was originally written for a much older version of Python than it now runs with. For example, I was always told that everything was an object in Python, so I was little surprised to see string being imported, and some methods being called from this module directly (I would have expected to see this as a method of the string object instance itself), but after reading up on things a bit more, it seems this is a throwback to the Python 1.6 days, so is a bit of legacy code.

I can't get used to the way slices work. I find the subscripting to be difficult to read. I've seen some code that removes the first and last character off a string if it's a period, and the way that did it still does my head in if I try to think about it too fast. I'm used to Java, as that's the only really object-oriented language I've learned, and so I'm used to the substring() method of string objects. I remember once I spent ages trying to figure out how to get a substring out of a string in Python. I've read Dive Into Python twice, and the first time I read it, I learned just enough to get myself into trouble, and I spent forever doing dir() on string variables in an interactive interpreter trying to find what Python called its substring method...

I can definitely agree with the argument that Python is more maintainable than say Perl. Perl's "more than one way to do it" mantra can lead to some horrible horrible code that not even the author can make sense of after six months. I bought Perl Best Practises a little while ago because the Jacinta and Paul from Perl Training Australia were giving it a good wrap, but I haven't actually had time to open it yet. Python's more single-tracked one of doing things tends to mean that the code is going to make more sense.

That said, the indentation thing is the pits. I miss curly brackets. I miss being about hit % in vi and jump between the start and end of a block. It's very easy to not realise when you've dropped off the end of a function definition, and of course if you move a block of code, you have to reindent. I don't doubt there's some funky vim stuff to deal with this, but until I figure out what it is, relocating chunks of code is going to be a bit tedious...

Other than that, it's not too bad. I did some serious hacking today, and got some results, so it wasn't all bad.

[22:09] [work] [permalink]

Tuesday, 07 March 2006


There is something really cool about being able to just ask my manager quick questions on IM...

[11:13] [work] [permalink]

Wednesday, 08 February 2006

Donated blood

Google does a lot of nice things for its employees, and its good to be able to pass that on when the opportunity arises.

Today, the American Red Cross visited the 'plex, and I made a donation. I think they did pretty well out of Googlers today, and it'd be cool if they published some statistics on how many donated, and how many litres/gallons of blood they got out of it, and some sort of statistic on how many people that is likely to help.

I haven't donated blood in Australia for ages. Come to think of it, I don't think I've ever donated in Canberra, it was too inconvenient, and the Canberra Blood Bank didn't seem to have the same sense of urgency about it that the Brisbane one did.

Certainly it was a very similar experience to donating in Brisbane. The usual questionnaire as long as your arm. The iron/red blood cell count. The main differences I noted were that they actually asked you all the questions in your interview, rather than asking you to fill out a form beforehand and going through any red flags with you. They also spun the pipette of blood to get a red cell count, rather than using copper sulphate and looking at the colour change (apparently it's a cost thing).

For the actual donation, they didn't use a rocker thingy like the Australian Red Cross did, they just let the bag hang there beside me while it filled.

Apparently they come out every four months or so, so I'll be happy to donate again given it's so convenient. I'd often mused about seeing if the Red Cross mobile donation unit could come to our workplace in Australia, but I doubted there'd be enough takers.

[21:04] [work] [permalink]

Sunday, 11 December 2005

You know you're working in a fantasy land when...

You say to your wife "Bleh. I won't need any money this week. The only thing I'd need money for is tipping the masseuse if I get around to booking a free massage".

[17:42] [work] [permalink]

Monday, 21 November 2005

Accidentally good choice of start date

We seem to have conveniently and unintentionally chosen a good week to start. Thursday is Thanksgiving, and a holiday, and Friday is as well, so three day week.

Talk about easing into things.

[22:53] [work] [permalink]

First day

My brain is full. I wonder if cranium extensions are covered by health insurance over here?

All I will say is that it is so cool, and that they have their induction process down pat, and the breadth and depth of their internal corporate intranet is nothing short of that of the Grand Canyon.

I am indeed feeling lucky.

That is all.

[21:20] [work] [permalink]

Tuesday, 15 November 2005

Roll back, or roll over and go back to sleep?

So, today is my last day at Cybertrust. Much to my displeasure it started at 4am, when I had to come into work to perform a load balancer upgrade.

I'd previously joked with a few people that I was tempted to not bother with the upgrade and just say I'd rolled back. Funnily enough, that was exactly the outcome anyway...

Being an uber secure facility, we keep all our class C rack keys in an electronic key safe. This being a funky PIN access-restricted, battery backed, solenoid-driven thing.

So when I rolled up at work at silly o'clock, the first thing I went to do was go to the key safe to pull out the rack keys I'd be needing to access the relevant racks to perform this upgrade. They key safe's little LCD display was dead. I laughed. I then proceeded to get lots of people out of bed to try and track down the location of the spare set of keys that wasn't in the key safe.

So to cut a long story short, we burned all our troubleshooting time trying to get rack keys, and we had some strange problems with locally attached devices not ARPing correctly, so we had to roll back anyway.

So I might as well have rolled over and gone back to sleep at 4am when the alarm went off. At least I'll get to knock off work early.

[15:01] [work] [permalink]

Friday, 21 October 2005

Just the way I want to spend my Friday night...

Bashing my head over why Firewall-1 is eating my ACK packets for dinner - when I'd rather be at home eating some myself. Dinner that is, not ACK packets. They're not all that filling. No payload and all.

So, the bastard thing has a rule that is supposed to accept packets from the big bad interweb, and let them in to a web host, after a spot of load balancing and what have you. I'm testing it with a remote connection from home. The SYN comes in, the SYN goes out. The SYN arrives where it's supposed to. The web server ACK's that SYN. The ACK arrives at the firewall. The ACK is never seen again. Oh, and the firewall logs an ACCEPT on the packet.

So after restarting Firewall-1, rebooting (gotta love the fact this isn't production yet), checking my routing (there's not a lot to check, it's going to go out the default gateway), I'm at headbutting keyboard point.

At least I found and fixed a problem with logging.

[02:42] [work] [permalink]

Wednesday, 12 October 2005

Blogging saves the day

Well kind of.

A colleague in Operations came up to me with a Firewall-1 problem, which was identical to one I'd experienced earlier in the year.

I couldn't remember the details, but I remembered the situation, so I just pulled up my blog, used Firefox's find-as-you-type feature, and pointed him at the details.

[19:03] [work] [permalink]

Tuesday, 20 September 2005

Oh the humanity!

So I dropped out of Uni this semester, and that included dropping out of a subject on concurrency, and here I am at work trying to deal with a problem of multiple scripts writing to a FIFO concurrently, and I'm trying to devise a solution whereby they'll only write one at a time, and also there won't be starvation.

Ye gods, the practical application of something at Uni. Who would have thought? But I dropped out... Argh.

[19:00] [work] [permalink]

Monday, 19 September 2005

There is no escape

So, I have been officially redeployed back to the client's site I was previously working in. So much for my little sabbatical at the office, and catching the bus... Ah well.

[18:05] [work] [permalink]

Thursday, 15 September 2005

Back at the office - the first fortnight in review

So, as I mentioned previously, my stint working on site at one of my company's clients ended at the end of last month due to contractual headcount reductions. I returned to the general Professional Services pool in at the office.

It's been an interesting couple of weeks, and I get the distinct impression that they don't really know what to do with me. This is partly of my own making, as I have been totally frank an honest with my management about an opportunity to work overseas that arose months ago. That has gotten to the point where I'm just waiting for the visa to get sorted out, and then I'm pretty much going to resign. Problem is, I have no real idea of when the visa is going to be ready, so therefore I can't give anyone, myself included, a better idea of time-frames than "probably November".

So for the first week and a bit, I sat at the desk of a project manager who was on leave, and did bits and pieces of a number of projects. The work was very stop-start though, and I found this a bit unsettling, as Professional Services is all about billable hours, and I didn't want to have big wads of unbillable time on my timesheet where I'd been twiddling my thumbs.

On Monday, when the project manager returned, I relocated to the "hot desk" - a desk intentionally kept vacant for visiting members of staff from other offices. That lasted until about lunchtime yesterday, when I was asked if I could help out with a stack of documentation relating to the old client whose site I used to work at, that needed to be done by the end of the month. So then I relocated upstairs to the NOC.

Then my old boss called me this afternoon and asked me if I'd prefer to go back to the client's site for a couple of weeks to help bail out a hardware upgrade project that had gone off the rails. I'd much rather to that than go batty trying to write gateway design documentation, so I agreed. I just have to wait for a new building pass to be issued.

So, in summary, I feel like I'm a bit... unallocated. But I guess that is the price I have to pay for being upfront and honest about probably leaving the company. I get to keep getting paid until I leave, and they get to throw me wherever suits them. C'est la vie.

[01:17] [work] [permalink]

Wednesday, 24 August 2005

Fun with xargs

Had some fun with xargs this morning. I came up with such a monstrosity, I have to record it for posterity.

The situation was one where we had a script that was scping a whole pile of files around, and the source directory got so big, that a straight "scp ${SOURCE}/* $DESTINATION" was resulting in the good old command-line too long situation.

Sounds like a job for xargs I say. But how do we tack the destination on the end? I'm used to situations where you just want to pass a whole lot of arguments to a command, but not have a constant value on the end. Oh, and this was on Solaris for good measure, so I was fully expecting to not be able to do it.

Did some prototyping. Put "a-z" in /tmp/alphabet, one per line.

apollock@caesar:~$ cat /tmp/alphabet | xargs -L 6 echo
a b c d e f
g h i j k l
m n o p q r
s t u v w x
y z

Right, so that solved the argument length issues, but I needed a constant argument on the end. I tried

apollock@caesar:~$ cat /tmp/alphabet | xargs -L 6 -i echo '{}' foo
a foo
b foo
c foo
d foo
e foo

But as you can see, (and so the manpage says), -i implies -l1 (which is the same as -L 1). Bummer.

So then I came up with this ripper:

apollock@caesar:~$ cat /tmp/alphabet | xargs -L 6 echo | xargs -i echo '{}' foo
a b c d e f foo
g h i j k l foo
m n o p q r foo
s t u v w x foo
y z foo

That's the ticket!

[17:31] [work] [permalink]

Tuesday, 09 August 2005

Last enforcement modules migrated

This morning and last Tuesday morning saw me in at work at stupid o'clock again migrating the last two enforcement modules in this site. Both went fine, and it's nice to have semi-completed a project. There's another site up in Sydney that I'm nominally supposed to also be migrated, but due to some contractual changes, I'm being pulled off this contract back into the general Professional Services practice at the office. Fine by me, a change is as good as a holiday.

[03:56] [work] [permalink]

Sunday, 05 June 2005

Configuring the timezone on a Cyclades AlterPath ACS

So I noticed that the timezone on these puppies at work was wrong, so I went about trying to fix it. I happened upon this page in German, which after translation (thanks Babel Fish!), I managed to decipher the following:

The factory default contents of /etc/TIMEZONE are:

and what I believe to be the desired contents for Australia/NSW or ACT:
From my understanding, translates to something like "The default timezone is GMT+10, but when we're in daylight saving time we want GMT+11. Daylight saving kicks in on the last Sunday of the tenth month at 2am and ends on the last Sunday of the third month at 3am." (So M10.5.0 means "day zero of week 5 of month 10). I presume this is a BusyBox thing. The things people do to save space.


It's a uClibc thing. Some good documentation on the TZ variable is at http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html


I swear drugs are involved here. I had to use

to get things to work as expected, which from my interpretation of some other documentation, really means I'm saying we're 10 hours behind UTC, which isn't the case, but yields a correct time. Go figure.

[17:22] [work] [permalink]

Thursday, 02 June 2005

Firewall-1: Great when it works, utter poo when it doesn't

Let me just start off by saying that CheckPoint Firewall-1 is probably my preferred EPLed packet-filtering firewall. The GUI is good, the fact that it is "object-oriented" is also good. What is not good is the complexity of a deployment and the ability to troubleshoot it when things go wrong. When it's broke, you are left with your pants down.

I've just deployed some new (non-production) firewalls. I've got a management station (which is also an enforcement module), which is multi-homed (5 interfaces). There are two enforcement modules managed by this management station, one on each of two of the 5 interfaces.

I'm able to push a policy from the management station to the enforcement modules. That's all good. If I try to make the enforcement module fetch a policy from the management station, it decides it would like to talk to it on one of the unrelated interfaces, rather than on the interface that is directly connected, or to the interface with the IP address of the management station object in the rulebase. Bonkers.

Furthermore, if I do issue a

fw fetch master
this works fine. It's just trying to log to the wrong interface of the management station, and fetch its policy (by default) from the wrong interface. Highly vexing stuff. As it's non-production, I'm getting close to reinstalling Firewall-1, but I wouldn't mind determining the cause for future reference in situations where this might not be an option.

[17:54] [work] [permalink]

Wednesday, 01 June 2005

Mmm Cyclades AlterPath ACS good

Disclaimer: I like Cyclades products. They Just Worktm

Yesterday and today I got to have a fiddle with the new AlterPath ACS product, and they're very special. I'd previously used the TS product a few years ago, and been quite happy with them, especially when having to use Digi's cheap imitation, that didn't Just Worktm.

I got LDAP authentication for individual ports working reasonably painlessly, and for good measure, then enabled local LDAP authentication to the terminal server itself, with similar ease.

The web GUI is a bit sluggish, but was still useable.

The userspace environment on the box was a bit stripped out for my liking, but still useable. A non-BusyBox vi would go a long way, as would iproute, but other than that, I really can't complain.

[18:55] [work] [permalink]

Sunday, 29 May 2005

Fun and games with cross-platform database interoperability

Okay, so I'm no longer buzzword-compliant with the current fashion on Microsoft data access. That's because I haven't followed it for around 5 years.

The problem:

We've got a Windows application (it's a honking great big CD burning "appliance" (I use the term loosely, it's really a full-blown Windows box with a burn of CD burners in it and a bit of robotic stuff going on)). We want to push files to it from a Solaris box, and have it produce CDs when it's received enough to fill a CD.

The software on the Windows box driving this whole process can apparently log to a database, using OLE DB. Good for it. We don't want to delete the files from the Solaris box sending them across until we know we've successfully got them on CD.

The current school of thought on how to deal with this was to use a MySQL database on the Solaris box, and query it to find out what files have been burned to CD, and then remove them from the Solaris box.

I got involved at the point where a co-worker had been bashing on some (what appeared to be) significantly dated piece of software to do OLE DB access to MySQL. I actually don't have the foggiest on what the difference between OLE DB and ODBC is. I'd previously had success getting Microsoft Access (as a front end) to talk to MySQL (as a back end) via ODBC, so I thought I'd have a fiddle.

I threw away the MySQL OLE DB stuff and got the latest greatest MyODBC driver for Windows (which appears to be significantly better maintained). I also grabbed the latest greatest Microsoft Data Access Components (MDAC) just to get the OLE DB provider for ODBC if I didn't already have it.

So the problem appears to be that the application, when it tries to create tables in the data source provided, tries to do it the "Microsoft SQL" way, and throws square brackets around the table name, and possibly the attribute names as well. Someone muttered something about part-time DBAs creating tables with spaces in the names.

I'm of the theory that this is happening in the application, and no amount of data access crap between the application and the database is going to clean this up. It's just not talking in an ODBC compliant manner. Damn it.

So the next approach is to try using MSDE as the database, and have it on the Windows box, and use FreeTDS on Solaris to query the database. Bags not having to build that.

I'd really like a sacrificial Linux box for prototyping. It would mean I could very quickly test the viability of things like this without having to inflict building the software on Solaris on someone (and waiting for it to happen).

[17:39] [work] [permalink]

Thursday, 26 May 2005

Who would have thought?

That on my favourite Unix, Solaris, you needed to have the NFS client packages installed in order to get the NFS server kernel module to load.

I love Solaris. Let's put bits of the kernel in random packages. That's the way.

[22:26] [work] [permalink]

Monday, 23 May 2005

World of pain (or 6th migration (fourth enforcement module))

Just when you thought it was safe to sleep past stupid o'clock...

I wrote last week that my firewall replacement program had been put on hold. Well, due to some internal politics, I ended up replacing the last remaining one in this particular gateway this morning, rather than leaving it with two arms on new firewalls and one arm on an old one.

I didn't actually mind too much, as it's kind of nice to actually vaguely finish something. I've only got two more in this site to do (in a separate gateway), but that'll happen probably in July at the rate things are going.

Anyway, I was in a world of hurt this morning performing this change, and it was all my fault.

I like to have a repeatable, auditable process for building these firewalls, and it's served me quite well to date. I think I need to more thoroughly check what I'm doing though. I did build this one in a bit of a rush because I only learned I was doing it last Wednesday afternoon, and did most of the configuration migration on Thursday.

The main source of problems was that I managed to migrate /etc/netmasks as /etc/defaultrouter. So not only did I bring up all my interfaces with completely insane netmasks, I also brought up my firewall with 8 default routes. This left me quite miffed, as I discovered after I'd swapped to the new firewall that I had completely bogus interface netmasks and a pre-migrated /etc/netmasks file. I couldn't figure this out, as I'd ticked off migrating /etc/netmasks on my checklist. It was only after a colleague came in and noticed the 8 default routes and that /etc/defaultrouter contained /etc/netmasks, that I realised what had happened.

I'm very annoying with myself for

  1. screwing up
  2. not realising pre-swap that I'd screwed up.

The moral of the story is that I need to actively recheck my checklist after I've completed it, not trust myself to have done each line-item correctly at the time.

At least there were no kangaroos this morning.

[17:07] [work] [permalink]

Tuesday, 17 May 2005

Fifth migration (third enforcement module)

This morning I did another Firewall-1 replacement. Not a lot to report. It was a bit more of an inconvience because I couldn't rack the new one in alongside the old one because of rack space constraints, so I had to start extra-early so as to get it all done before the testers all did their thing.

The remainder of the rollout has been put on hold as I'm being transferred over to a disaster recovery project (well, it's really a project to build a fully DR-capable replica of the existing site for deploying up in Sydney).

At this stage it's just for June, but who knows how long it'll really be?

[00:01] [work] [permalink]

Friday, 06 May 2005

I have a green helmet

No, I don't have a gangrenous penis.

Today, I was issued with my helmet as I had passed my first aid course.

For some reason, first aid officers get a green helmet. Chief fire wardens get a red helmet, floor wardens get a yellow one.

I'm not quite sure why someone who performs first aid requires a helmet. If I don't wear it, can I still provide first aid?

[01:47] [work] [permalink]

Monday, 02 May 2005

Fourth migration (second enforcement module)

This morning I got up at a quarter to 5 to get into work and do the next firewall migration. This one had to be done "out of hours" because I'm getting into the territory of the firewalls that see all the action now.

I was really apprehensive about this one, given that I first started building this firewall about a month or more ago. With a lot of politics, hardware orders taking forever to turn up, and me taking a couple of weeks off to help run a conference, the schedule got pushed back a lot. It had been so long since I'd done a migration, and so long since I'd built this firewall, that I was really worried I'd mess something up.

Fortunately all my checklists I'd made seemed to hold up alright. Yay for quality assurance. Some strange anti-spoofing problems crept in, but a co-worked is going around doing a general anti-spoofing cleanup after me, so I've left that for them to resolve in a more permanent fashion that I did.

[21:27] [work] [permalink]

Wednesday, 16 March 2005

Adventures in reverse engineering

This week has been fun. I've been reverse-engineering how a a Linux-based load balancing appliance works.

The appliance is an F5 BIG-IP Local Traffic Manager.

Up until version 4.5, they used to be BSD-based, but they went to a new hardware platform, and decided to double 4.5 and came up with v9, which incidentally, appears to be Red Hat 9 based.

We want to be able to customise the build process so that we spit out a site-specific-configured BIG-IP. No problem I think, I'll just build an RPM containing all of our config files in it. I'd previously pulled to bits the installation process, and it was quite trivial to just grab the ISO, unpack it, chuck an extra RPM on, add that RPM's filename to a file, and rebuild the ISO. Hey presto, my "Hello, World" proof of concept RPM was being installed on a BIG-IP.

So then I tried to go for gold, and built a preliminary config RPM, with our password file in it and whatnot. This is where I got too clever for myself and forgot one minor problem. Half the files I'm trying to install in my RPM belong to other RPMs already installed, so of course RPM bleats, and the package doesn't install. Bummer. I need to find out if I can declare that one RPM overwrites bits of another one, otherwise I'll really have to hack to the installer so that it can force a specific bunch of RPMs in.

<rant> It would help if I could find a canonical, current source of documentation for the RPM spec file and RPM building in general. Google is useless. You put "rpm" and "spec" together, and you start finding all sorts of random spec files for packages, which is not what I want. www.rpm.org is grossly out of date, and not terribly in-depth, and the chapter of the Fedora Developer's Guide is a joke. </rant>

[14:46] [work] [permalink]

Wednesday, 09 March 2005

Enforcement module migration SNAFU redux

Just when you thought it was safe to push a firewall policy...

Today one of the Operations guys tried to push an updated policy to the enforcement module that I migrated recently and was greeted by some errors regarding "No valid FM license". (I still haven't figured out what FM stands for yet).

I've no idea why this happened out of the blue. I could certainly push a policy after I finished the migration. I restarted Firewall-1, and also received some "No valid FM licenses" during the initialisation messages.

I pulled up the SmartUpdate application, and detached the licenses associated with that node and reattached them (well I noticed that one of them was for an IP address that wasn't on that firewall so I left that detached) and did a cprestart, and everything came good. I gave it a reboot just to make sure it wasn't going to return to SNAFUness after a reboot, and it was still good.

I look forward to the next enforcement module migration with much fear and trepidation.

[21:08] [work] [permalink]

Monday, 28 February 2005

Third migration (first enforcement module) SNAFU

Yesterday, I migrated my first actual enforcement module. What was supposed to be quite simple, went quite pear-shaped instead.

Fortunately, I picked a relatively unimportant firewall for the first cab off the rank, so the fact that I ran an hour over the alotted change window wasn't an issue. It also enabled me to keep bashing on the problem until I resolved it, rather than having to back out.

What was the problem? Well, it was actually a problem with the migration of the management server for that particular enforcement module. When I migrated the SIC (that's Secure Internal Connection for you non-Firewall-1 savvy people) related crap in $CPDIR/registry/HKLM_registry.data, I screwed up, and didn't set the 6 characters in the SIC's distinguished name to the same thing for both occurences in that file, which produced quite screwed up results when resetting the SIC between the management server and the replaced enforcement module.

What I had was:

: (SIC
        :ICAState ("[4]3")
        :ICAdn ("o=my_management_server..yyyyyy")
        :HasCertificate ("[4]1")
        :MySICname ("cn=cp_mgmt,o=my_management_server..zzzzzz")
        :CertPath ("/opt/CPshrd-53/conf/sic_cert.p12")

when I really should have had:

: (SIC
        :ICAState ("[4]3")
        :ICAdn ("o=my_management_server..zzzzzz")
        :HasCertificate ("[4]1")
        :MySICname ("cn=cp_mgmt,o=my_management_server..zzzzzz")
        :CertPath ("/opt/CPshrd-53/conf/sic_cert.p12")

This had the interesting effect of the enforcement module getting the 'zzzzzz' SIC during the initial SIC initialisation, but the management server thinking it was 'yyyyyy', and expecting this during normal SIC operation, so nothing worked.

This problem hadn't manifested itself for the other enforcement modules, as they must only deal with the 'MySICname' part of HKLM_registry.data for normal operation. I'm guessing the 'ICAdn' is only consulted when the SIC is reset.

So I just fixed up the HKLM_registry.data file on the management server and restarted Firewall-1 on it, and then lo and behold, I could establish a connection to my new enforcement module.

[16:55] [work] [permalink]

Monday, 21 February 2005

Second migration successful (well, kind of)

This morning I did the second (and final) Firewall-1 management server migration at this site. It wasn't as successful as the last one, in that it didn't Just Work.

With some help from Jonathan, the problem was traced back to the Get Topology function getting it wrong. I have to do a Get Topology after I've migrated the configuration as the new hardware has different Ethernet device names to the old one (gotta love how Solaris has hardware specific Ethernet device names). Unfortunately, in the process of doing the Get Topology, Firewall-1 decided to mark one of the interfaces as External, when it really should have been Internal, so then the anti-spoofing stuff kicked in and it decided that connections that were legitimate were actually spoofed, and dropped them.

It made matters worse (but was probably a blessing in disguise in that it highlighted the problem immediately) because the interface in question was the one that connected this management server to the rest of the management network, you couldn't get through the management server (which is also an enforcement module) to other hosts behind it.


It wasn't so much a case of the Get Topology function getting it wrong. It seems that Firewall-1 will assume that the interface with the default route going out it is external. So for this particular firewall, I just need to redo the routing so there are specific routes and no default route, and in theory everything should be considered internal.

[17:10] [work] [permalink]

Thursday, 17 February 2005

Oh no!

The dude with the disturbing ringtone has just moved directly behind me.

[16:33] [work] [permalink]

Monday, 14 February 2005

Learning awk

As bizarre as it my seem, I've managed to get through life until this point without knowing any awk. If cut and paste didn't cut it (no pun intended), I'd just write a Perl script and be done with it.

There's the possibility that I might have to maintain some behemoth monstrosity of an awk script, so I'm using a bit of spare time whilst the bureacratic wheels turn to read O'Reilly's sed & awk, Second Edition

[17:36] [work] [permalink]

Wednesday, 09 February 2005

Pride in your work

The convoluted way things work (for procurement) within the client's organisation I'm working at are that all hardware is ordered through, and remains property of EDS.

So for this firewall replacement project I'm doing, a bunch of hardware (mainly V240's) was ordered. Another project ordered a few V440's with fibre channel cards. Sun being Sun, ship the cards separate to the boxes, and EDS fit them. Problem is, EDS didn't consult the order when they went to install the cards, and just saw that there were the same number of cards as boxes and installed one in each (when in fact the V440's should have had two cards each and the V240's none).

So this morning, I proceeded to remove the fibre channel card from my V240 for the next firewall I'm replacing, and put it into one of the V440's. I had to pinch a blanking plate from the V440 to fill the gap in the V240. All good, I think. Wrong.

I get the lid off the V440, and discover that the existing fibre channel card hasn't been screwed in, and the blanking plate for the (vacant) PCI slot next to it is (poorly) held in by an ill-fitting PC case screw. Looks like they lost some screws when they were installing the cards methinks.

The V440's have a mix of 33 and 66 megahertz PCI slots. The only cards to go in the box are the two fibre channel cards, and of course EDS has installed the one that is already in there in a 33 megahertz slot, when they had the pick of the slots. So I moved the existing card while I was in there, and installed the one I took out of my V240 in another 66 megahertz slot, found a random screw that fit so that both cards were screwed in, closed it up, and thought I'd have a bit of a rant about taking pride in your work.

[17:45] [work] [permalink]

Tuesday, 08 February 2005

Windows Update is stupid

So I fire up Internet Exploder to run Windows Update (I only use it for this and submitting my timesheet) so as to download this month's plethora of critical updates for Windows, and the stupid thing wants to ignore my proxy settings and make direct connections for the downloads. This of course won't work, so the downloads fail. Nevermind the fact that I used a proxy server for every HTTP connection up to the point of initiating the downloads.

[16:04] [work] [permalink]

Loopback devices under Solaris

I was helping a co-worker grok how to loopback-mount an ISO image, and after discovering it was lofiadm that one needed to use in place of losetup, was having a bit of a peruse of the manpage and was amused no end to note that the examples they are are mounting an Red Hat Linux 6.0 for SPARC ISO image.

You wouldn't expect to find this subtle reference to a competing operating system within the Solaris documentation.

[15:49] [work] [permalink]

Sunday, 06 February 2005

First migration successful

So this morning I migrated my first Firewall-1 management server in the production environment. Thanks to lots of testing and experimenting and breaking and fixing things in the test environment, I pulled this off without a hitch. And the management servers are the hard ones. The enforcement modules are a piece of cake. There's nothing to migrate except the license. I just need more hardware to turn up before I can proceed further. In the meantime, I've scored another project to do on the side, migrating a data service from an old firewall environment to a new(er) one.

[16:39] [work] [permalink]


My boss' boss has agreed to give me two hours a week paid study leave to attend University classes. The rest I have to make up myself. Better than a kick in the teeth, and better than having to make up the total contact hours myself. Now if I can just get tutorials outside of work hours, I should only have to make up about three hours a week of lectures.

[14:59] [work] [permalink]