Sunday, 04 January 2009

(or echo "blosxom hold" | sudo dpkg --set-selections)

A few people had pointed out to me that my blog was appearing weirdly in Google Reader. I myself read my blog via Planet Debian and Planet Linux Australia, where it appears fine, so I hadn't noticed the problem first hand.

I did some digging tonight with the Feed Validator, and I think I've figured it out.

Funnily enough, it all started with a previous session with the Feed Validator...

I wanted my blog to be UTF-8, with it being the new hotness and all. Blosxom was spitting it out as ISO-8859-1. I've just learned from this evening's debugging, that this is the behaviour of CGI.pm's header() function if not provided with the -charset parameter.

I'd previously worked around this by defining my own content_type for the RSS and HTML flavours. In this file, I just put text/xml; charset=utf-8and text/html; charset=utf-8, respectively.

Unfortunately, this then breaks Blosxom's internal HTML-escaping-for-RSS-feed code, which is conditional on the content type being XML:

if ($content_type =~ m{\Wxml$}) {
        # Escape <, >, and &amp;, and to produce valid RSS
        my %escape = ('<'=>'&lt;', '>'=>'&gt;', '&'=&gt;'&', '"'=>'&quot;');  
        my $escape_re  = join '|' => keys %escape;
        $title =~ s/($escape_re)/$escape{$1}/g;
        $body =~ s/($escape_re)/$escape{$1}/g;
      }

This block of code was being ignored because the regex was anchored. "text/xml; charset=utf-8" didn't match "\Wxml$". Whenever I last had an attempt to make my blog's feed pass the Feed Validator, I must have edited the Blosxom source code to make this regex not be anchored.

Then on November 11, 2008, I updated my Blosxom package to 2.0-14+etch1, to address CVE-2008-2236 (incidentally, there doesn't appear to be a Debian Security Advisory for this?), and my local modification got clobbered. That'll teach me to make local modifications and not put the package on hold. Blah.

It seems I'm screwed, there's no way to have a well-formed UTF-8 RSS feed without requiring a source-code modification to Blosxom. I've either got to change the way the header() function is being called, or I've got to change the regex to allow me to specify a charset in the flavour file, and not disable HTML escaping.

Speaking of screwed, Blosxom seems incapable of making a well-formed RSS 0.91 feed anyway. It wants to put the <pubDate> tag inside an <item>, which is a violation of the DTD. There's supposed to be one for the whole <channel>, not one per item (or "story" in Blosxom-flavour-speak). You can't just move that from the story flavour file to the head flavour file, because a pile of variables that you'd otherwise use seem to only get set on a per-story basis. Double blah.

I'm starting to feel like Blosxom is outliving its usefulness, but I really do not feel like embarking on a wholesale move to a new blogging platform, I'm fairly entrenched.

(this is something I've had "in production" for many months now, I just haven't had the time or energy to do a proper write up about what I did)

We have a cat water bowl, it looks like this:

Under "normal" circumstances, it usually lasts about seven days. So when our weekly routine is happening, we'll refill it on a Saturday whilst doing house chores, and it'll last until the following Saturday when it gets refilled.

Unfortunately, sometimes our routine gets disrupted, and we forget. Sometimes, we travel and have a house-sitter, who may not pay as close attention to such things as ourselves.

Once, one of our cats was licking the condensation off a chilled bottle of soft drink that was on the kitchen counter one evening, before we realised the water bowl needed refilling. Naturally, we felt like terrible pet owners.

So I think it was some time around the 2008 Maker Faire, that I hatched the idea of having some sort of water sensor on the cat bowl, which would communicate to one of the various computers in house. At the Maker Faire, I bought a copy of Making things talk, and an Arduino starter kit, which consisted of a Diecimila board and a make-it-yourself proto shield. I also bought a little electronics starter kit, which consisted of a breadboard and various components, and a USB-TTL cable.

I decided to use Bluetooth to communicate with the board, as I already had my MythTV setup using a Bluetooth keyboard and mouse, and it was within range of the water bowl. I decided against using Zigbee, because I didn't know anything about it, and I didn't want to add (or learn about) yet another wireless infrastructure just for this project.

I should point out that I know very little about electronics. I'd never owned (or really used) a soldering iron until I embarked on this project. I took a basic soldering class at the Tech Shop, but I'd already assembled the proto shield by the time I took the class, so I'd pretty much figured it out.

I had a very naive vision that I could just basically shove two wires in some water and it'd close a circuit and that would be my water sensor. Of course this didn't work, so I started hunting around on the Internet for a circuit that would do this. I happened upon a circuit (I don't seem to have retained the URL, so I can't link to it), which just consisted of a couple of transistors and some resistors. So I headed off to Fry's to try and buy the transistors I needed.

I quickly discovered that I didn't have sufficient information to identify the transistors that I wanted, but I did happen to stumble upon a cheap assemble-it-yourself water alarm. It consisted of a PCB, and a transistor and some resistors and a buzzer. I bought a couple of these instead.

Between studying the PCB and the circuit diagram that came with the alarm, I was able to reproduce it on my breadboard instead of on the PCB. Sure enough, placing the two probes in water closed the circuit. I replaced the buzzer with an LED so I could see what was going on.

I attached the circuit to the Arduino proto shield, and had it feed into one of the digital I/O ports. I wrote some quick and dirty Wiring code so that when water not present (i.e. the circuit was open and no current was detected on that I/O port) the LED was switched on. Really at this point, I didn't need a microcontroller, I could have presumably achieved the same thing with a NOT gate.

At this point, I wanted to make the sensor remotely queryable. I bought a BlueSMiRF Silver Bluetooth modem, which I attached to the TX and RX lines of the board (I first configured it by attaching the USB-TTL cable to it and using Minicom on my laptop). I extended the Wiring code to provide a rudimentary prompt, and accept a command to check if water was present.

I think around this time it also dawned on me that I could use the digital I/O pins as a switch. When they're "high" they provide power. So rather than constantly running a current through the water, I only needed to briefly power up the water detection circuit, see if the circuit closed or not, and then report if water was present if it did.

I much preferred this, as at the time, I was endeavouring to power the whole sensor off a 9 volt battery. I figured I'd get much better battery life if I wasn't running a current through the water the entire time. I should point out that I did some "tongue tests" in a glass of water while the circuit was powered up, and couldn't detect a difference between when the circuit was on or off. The last thing I wanted to be doing was zapping the cats!

At this point, the Wiring and Arduino work was pretty much complete. I setup ser2net on the MythTV server, so that I could just connect to port 4000, and be connected to the water sensor.

apollock@icarus:~$ telnet teevee 4000
Trying 172.16.0.9...
Connected to teevee.andrew.net.au.
Escape character is '^]'.

ser2net port 4000 device /dev/rfcomm0 [9600 N81] (Debian GNU/Linux)


waterbowl> s
Water is present
waterbowl>
telnet> q
Connection closed.
apollock@icarus:~$

The Wiring code running on the Arduino board is checked in here.

Next, I wanted to monitor this with Nagios. One thing I found with the Bluetooth connection was that it wasn't all that reliable. Not every connection to port 4000 resulted in a connection with the water sensor. I elected to write some standalone code that submitted results to Nagios by way of a passive check, rather than having Nagios try to actively monitor it. Again, trying to conserve energy, I decided to only check the sensor once every 8 hours.

So I wrote a Python daemon, which tried to be as robust as possible. If at first it doesn't make a connection on its 8 hourly schedule, it keeps trying until it does. Nagios itself has some freshness detection for this monitor, so if no passive result is submitted within 8 hours and one minute, Nagios alerts that something is possibly wrong with the sensor itself. (The original intent was to deal with the situation where the battery went flat)

This is the Nagios service definition I've got. Some of it may be unnecessary or redundant:

define service {
        host_name                       teevee
        service_description             Cat water bowl
        check_command                   check_stale!2!"Check water bowl monitor is on and reachable"
        normal_check_interval           480
        notification_interval           240
        active_checks_enabled           0
        check_freshness                 1
        freshness_threshold             28860
        max_check_attempts              1
        check_period                    24x7
        use                             generic-service
        stalking_options                o,w,c
        contact_groups                  everyone
}

The code for the Python daemon is checked in here.

So everything was going swimmingly, apart from I couldn't get even 24 hours of monitoring on my conservative 3 times a day schedule, and the Bluetooth modem configured for all of its power saving options, and the water sensor itself only being powered on when it was performing a check. There wasn't much more I could do to try and reduce power consumption, at least with my limited electronics knowledge. Perhaps using Zigbee instead of Bluetooth would have helped, but I still think I'd have been going through 9 volt batteries more quickly than I'd have liked.

I also had a brief foray into solar powering the board. I thought maybe the lighting in the house would be sufficient to power the board. Of course this didn't work out. I'd also have needed to make the code running on the board more self-sufficient, and have it just provide a water status indication whenever it had enough power to do so, instead of being a fairly dumb device like it currently is. This all felt a whole level more complicated, and out of my league, and I wasn't interested in attempting this sort of remote-sensing exercise at this time.

The Arduino board can also be powered by USB, so as I already had some long USB type A to type B cables (that had funky lights in them to boot), I decided to just get a wall wart that had a USB type A receptacle, and powered the Arduino board that way. So much for being completely "wireless". (I could have also gotten a general purpose DC power supply that was capable of spitting out 9 volts, but I doubted I'd get one with a sufficiently long cable, which is why I went for the USB-powered option, as I already had a cable long enough)

Speaking of wires, the most challenging part was the probes for the water sensor. As they were going to be permanently in the cats' drinking water, I didn't want to contaminate the water with them. I figured plain untinned copper wire would be okay, since water pipes are copper. Finding untinned, unstranded copper wire was a real challenge. I started out using some FM antenna cable, but that was stranded, and the shielding was a nightmare to strip. It was also reasonably difficult to make go where I wanted it.

What I really wanted was something more solid, that would flex and stay where I put it. I cannibalised a spare IEC power cable, but it was also stranded copper wire. I finally managed to obtain some solid-core CAT-5 cable from a hardware store, and this has worked exactly as I wanted.

I haven't done any further work on the setup since getting the CAT-5 cable for the probe. Further improvements that I'd like to do at some point:

• transfer the circuitry from the breadboard to a breadboardless proto shield, and generally try to house the sensor better
• find a better way of connecting the probe wires to the circuit, so it's easier to remove the monitor when refilling the water tank (currently the CAT-5 wires are just shoved into the breadboard). The whole thing is barely spouse-approved as it currently is, let alone being easy enough for an untrained house-sitter to try and negotiate
• integrate with Asterisk so a house-sitter gets a telephone call when the water bowl needs refilling

I had a lot of fun with this project. I had a real sense of achievement, being able to go from concept to completion, and learn a few things about electronics along the way. I'm normally not a fan of messing around with hardware.

Some photos of the project are here.