Diary of a geek

March 2010
Mon Tue Wed Thu Fri Sat Sun
16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31        

My ugly mug

Where's Andrew?

Categories

Other people's blogs

Subscribe

RSS feed

Contact me

JavaScript required


Sunday, 07 March 2010

Backspacegate

I just updated to the latest beta of Chrome, and the backspace key stopped working as a keyboard shortcut for the Back button.

After a few times of stabbing the backspace key and not getting the result I wanted, I decided to go looking into what was going on here.

It looks like it all started with bug 30699, where someone didn't like the default behaviour. That led to bug 36533, when the people (like me) noticed the functionality they were relying on disappeared.

Now I fully understand that Backspace == Back is not the default behaviour of Firefox (on Linux), but it is a configurable option, and I'd had it enabled there for years. I think it all started with when I migrated from Windows to Linux. It's normal for Backspace == Back with IE and I think Firefox for Windows, and I've just developed the muscle memory for it, and I've never had a problem like what the submitter of bug 30699 was complaining about.

I look forward to it becoming a configurable option in Chrome.

[18:51] [tech] [permalink]

Wednesday, 20 January 2010

Finding the maximum message size

This was born from a need to see how big a ZIP file I could send my accountant in Australia, and scratching the itch to write some code.

The fact that most SMTP servers talk Extended SMTP makes this relatively easy, and Python has some great modules for DNS and SMTP.

One gripe I do have is how long it takes the Python modules to mature. It's taken until Python 2.6 for smtplib.SMTP() to gain a timeout parameter, for example.

Anyway, I was able to write something nice and generic (it works for any domain) in around 100 lines, thanks in no small part to the DNS module, which makes getting a list of MX records stupidly easy.

$ ./maxmessagesize.py andrew.net.au
daedalus.andrew.net.au: -1
$ ./maxmessagesize.py pollock.id.au
ASPMX.L.GOOGLE.COM: 35,651,584
ALT1.ASPMX.L.GOOGLE.COM: 35,651,584
ALT2.ASPMX.L.GOOGLE.COM: 35,651,584
ASPMX2.GOOGLEMAIL.COM: 35,651,584
ASPMX3.GOOGLEMAIL.COM: 35,651,584
ASPMX4.GOOGLEMAIL.COM: 35,651,584
ASPMX5.GOOGLEMAIL.COM: 35,651,584
$ ./maxmessagesize.py debian.org
master.debian.org: 104,857,600
$ ./maxmessagesize.py cameronp.com
mail1.mysmarthost.com: 30,000,000
mail2.mysmarthost.com: 30,000,000
$ ./maxmessagesize.py ubuntu.com
mx.canonical.com: 62,914,560
$ ./maxmessagesize.py clug.org.au
mx.clug.org.au: 50,000,000
$ ./maxmessagesize.py linux.org.au
morton.linux.org.au: 52,428,800

It's good to see that in most cases of domains I tried, all of the MXes had the same maximum message size.

Source code is here

[23:51] [tech] [permalink]

Sunday, 29 November 2009

Review: iGala digital picture frame

Now that I'm not letting the cat out of the bag (I bought these as gifts) I can write a review.

The use case was pretty simple: I thought it'd be a cool Christmas present for our parents to give them something Internet-enabled that would give them regular updates on their grandchild (when he/she arrives). I was thinking a picture a day type of thing.

So I hunted around for a WiFi-enabled digital picture frame, and found the iGala being sold pretty much exclusively by ThinkGeek.

The reviews seemed pretty good. I know WiFi-capable frames have been around for a few years, but they always seemed to be pretty lacking in terms of WiFi functionality. Like they wouldn't do any security, or they'd only do WEP. This particular product claimed to do the whole gamut, including WPA2. The fact that it was a touch screen and ran Linux also made it appealing.

So I ordered a couple of them for a few weeks before we headed to Australia, with the intention of making sure that they'd work. Here's the highlights.

WPA2 didn't work (nor did WPA)

Despite the software on the frames claiming to be able to talk WPA2, the frame would not associate with my Linksys WRTU54G-TM. I had to drop it all the way back to WEP to get it to connect. For me, this was the most disappointing failure. I bought the product specifically on the strength of its claim that it supported WPA2, and it just didn't work. It was also pretty impossible to debug the failure.

I downloaded the latest firmware update, and that added additional settings for TKIP or AES when selecting WPA2, but neither option helped.

The automatic updates are brain-dead

Speaking of downloading firmware updates, the latest firmware that I downloaded and installed on the frames added automatic over-the-air firmware updates. Nice enough feature, except for the implementation. The frame tried to make an HTTP GET request for a non-existent file, every 6 seconds.

So the frequency of checking alone is totally ridiculous, but couple with this the fact that it's making a GET request (this is what $DEITY invented the HEAD request for, people!) and the website has a "friendly" 404 Not Found page that weighs in at a little over 10 Kb. By my calculations, that's nearly 150 Mb of failed update checking traffic a day. Taking these frames to a backward country like Australia, where ISP users still have monthly quotas, gives the frame a pretty horrendous running cost in terms of traffic. Not to mention the outbound bandwidth requirements for the server hosting the updates. Crikey, the mind boggles.

I'd have thought checking once a day and on power on would be perfectly sufficient.

Transitions are unavoidable

It may be just me, but I hate cheezy transitions. Digital picture frames tend to come with a myriad of them, but they all look cheap. It's impossible to tell the frame to just change the picture, it has to use at least one transition effect all the time. It defaults to randomly choosing from all the available ones. At least you can tone it down to just one.

Automatic on/off time

I liked that it was possible to configure operating times. No need to have the thing chewing power 24x7. It just seems to turn off the backlight outside of the programmed operating hours, so it's still doing the lame uber-frequent and bandwidth-intensive checking for updates even when it's "off".

Photo check frequency is configurable

Another nice feature was the ability to check for new photos at varying intervals. What I wanted for my parents was to just update once a day, so they'd get a new photo every day (assuming we put a new one in the Picasa web album that it's checking). This was very doable, and coupled with the automatic on/off time, means they should wake up to a new photo every day (that we change it).

Built in photos are a bit too sticky

There's 3 or 4 in-built photos as part of the firmware. If there's nothing accessible or available online, it'll cycle through these. Somewhat annoyingly, you need to have at least two photos in your online source for it to stop wanting to incorporate the stock photos in the mix. The workaround is to put the same photo in the online album twice, so you don't realise it's switching between two images. Lame, but it works...

Touch screen UI was adequate

Given the alternative user interface for digital picture frames is a little IR remote control and some dinky menus, the iGala was nice to configure. A full on-screen QWERTY keyboard pops up for entering WEP/WPA/WPA2 keys and configuring the Picasa/Flickr connections.

Fairly responsive support

The main near-showstopper for me was the lack of advertised WPA2 support. I emailed the Aequitech support folks quite a bit during my "evaluation" period. They got back to me fairly quickly most of the time and wanted to know exact details of my setup so they could reproduce it in the lab. They'd be well served having an actual ticketing system, instead of hiding behind an email address, as it made it hard to keep track of the multiple issues I was raising with them.

It's written in Lua?!

I have no familiarity with Lua, other than I know of its existence as a programming language. I'm curious as to what their motivation was for this language choice. All of the Lua code shipped in the complete firmware refresh ZIP files is bytecode. I have no idea if it's possible to decompile it. The CPU architecture would appear to be a Blackfin based on the few compiled binaries included in the full firmware.

Easy to update

Prior to the new update "functionality" I've already railed against, it was pretty easy to update. Download a ZIP file and a shell script, put them in the root directory of a USB key, and plug it into the frame and stand back. The updates don't seem to cryptographically verified (even the over-the-air ones), so I wonder if it's possible to break into the frame by way of a cleverly crafted "update". I have no idea what breaking into the frame would buy you. I don't know what sort of computing power they have.

Conclusion

I still think the iGala is a reasonable, if somewhat immature product. If the software is going to be actively worked on, and the support people continue to be responsive, then I think it's got good potential. For the price, I expected a more polished product, though.

I received an email from their support people shortly after returning from Australia saying that they'd fixed the WPA2 problem. Unfortunately I had no intention of trying to remotely talk my parents through how to reconfigure their access point or the frame (interestingly WPA2 didn't work with their Linksys WAG54G2 either, so I'd love to know what WPA2 devices it was tested with), so it's 128-bit WEP until I next go to Australia.

[22:57] [tech] [permalink]

Friday, 27 November 2009

mirror.linux.org.au upgraded

I took advantage of the four day weekend for Thanksgiving, and finally got around to upgrading mirror.linux.org.au from Debian Etch to Lenny.

The upgrade went fairly well. Notably, Drupal completely blew up, but it looks like we were still running the package from Sarge, as Drupal wasn't in Etch at all. I cut my losses, installed Drupal 6 and put something basic together from scratch.

MoinMoin upgraded fairly painlessly, and I figured out how to fix Cacti for my installation at home at the same time, so that was a general win all round.

[18:21] [tech] [permalink]

Saturday, 21 November 2009

LVM gaining the ability to merge snapshots

I love LWN. It's the best value for money way of keeping abreast of what's going on out there.

I also love LVM. I'm thrilled to learn from a comment on this article about Btrfs, that LVM is soon to gain snapshot merging support.

This is going to be absolutely fantastic for rolling back upgrades that go bad. I can't wait.

[09:52] [tech] [permalink]

Friday, 20 November 2009

"#!/bin/sh -e" considered harmful

Russell Coker advocates putting -e on the shebang line of shell scripts.

I disagree. From my experience this is extremely unhelpful to people who may be debugging your shell scripts in the future.

Consider this, you've added -e to the shebang of a script, and some poor schmuck down the track is trying to debug why it spontaneously exits. What's the most obvious way to do this? Run the script with sh -x or bash -x.

What happens when you do this? The shebang is completely ignored, and the script is directly run by the shell interpreter. If the person doing the debugging doesn't happen to transpose all of the shell options on the shebang line to the manual shell interpreter invocation, you're going to get different behaviour.

So I advocate an explicit set -e on the second line of shell scripts instead.

As much as making shell scripts set -e is a good practice, it drives me absolutely batty having to deal with scripts that spontaneously exit as soon as something they run exits non-zero. Particularly when you've chained a bunch of shell scripts together, or have one sourcing a bunch of script fragments from a directory. For this reason, I prefer to write in Bash and use an exit handler, to make it very obvious when a shell script has abended due to set -e.

[10:04] [tech] [permalink]

Sunday, 30 August 2009

Finding out about conferences

Paul Fenwick says he finds out about conferences by word of mouth.

Not that I ever do more than skim the page, but the Announcements page of Linux Weekly News (a fine publication well worth more financial support) mentions upcoming conferences. I've just learned that this page is derived from the LWN.net Community Calendar

[21:52] [tech] [permalink]

Monday, 17 August 2009

Having your cookie and eating it too

Russell Coker seemed to be of the impression that Firefox lacks support for manual cookie acceptance, whereas Konqueror has it.

Never fear, Russell! They just hid it very well:

Firefox 3.5 Privacy/History preferences

I must admit I had to do a bit of digging to find it, but I just couldn't believe that they'd take away a feature like that.

[21:07] [tech] [permalink]

Saturday, 08 August 2009

fit-PC2

The fit-PC2 looks interesting. As say a MythTV front end, I think it would be more promising than a Zonbu Mini, at a similar price. It certainly looks sexier.

If my current front end wasn't also a back end and therefore had a cable tuner card in it, I'd be more interested in exploring this. That said, I could investigate more seriously discontinuing using the cable tuner card and just using the HDHomeRun for all my tuning needs. Then my MythTV setup would be crazy distributed.

In fact, for my own edification, let's do a side-by-side comparison...

 Zonbu Minifit-PC2
Price$299 USD$359 USD
Processor1.2 Ghz VIA C7-M1.6 Ghz Intel Atom Z530
GraphicsVIA CX700M2, VGADVI
RAM512 MB1 GB
WiFi802.11bg optional extra802.11bg standard
Storage4GB Compact Flash160GB 2.5" SATA

If you opt for the diskless one for $245 you get a 1.1 Ghz Intel Atom Z510 instead. Not sure if the WiFi chipset is different as well. So if you scrap the storage, you can undercut the cost of a Zonbu Mini considerably. I'm not sure how the Atom compares with the VIA CPU in terms of actual performance.

Interestingly, it seems if you want a device capable of automatically booting when power is restored, you have to order a specific model, the "instant on" one. Sheesh. You'd just make that a BIOS setting and be done with it.

It sounds like the display is something proprietary. Pain.

Sadly the miniSD port doesn't seem to support SDHC, so I guess you'd have to use a USB stick if you wanted to go for a non-rotational-disk-based solution.

It's good that they're up-front about all of this.

Despite some possible short-comings, there's something very appealing about a device small enough to be able to bolt onto the back a TV using a VESA mount, although that would make the integrated IR received kind of pointless, which is a shame. This looks like a device worth keeping an eye on.

[16:37] [tech/gadgets] [permalink]

Friday, 07 August 2009

It's nice to be appreciated

I've been a volunteer sysadmin for Linux Australia since some time after linux.conf.au 2005

My enthusiasm (and therefore output) has waxed and waned over this time period, but to my totally unexpected delight, on System Administrator Appreciation Day this year I received a $100 ThinkGeek gift certificate.

So that was nice. Now I'll just have to work on my motivation...

[22:08] [tech] [permalink]

Sunday, 12 July 2009

I can stop going crazy using Chromium on Linux

Chromium bug 13395 has been fixed. All is right with the world.

[11:14] [tech] [permalink]

Thursday, 18 June 2009

Shout out to SiliconDust

I bought a HDHomeRun at some point in the past. It works ridiculously well with MythTV and helps add to the distributed network that is our home.

Recently, the power supply seemed to give up the ghost. The little red light on it was flickering on and off, and the unit became unresponsive. On the weekend, I finally got around to filing a ticket with SiliconDust to enquire if I could purchase a replacement, because I figured the entire unit was out of warranty by now (I said as much in the ticket).

To my surprise, I received a reply telling me that based on my device ID, I had a known-bad power supply, and I was directed to fill out an online form to get a replacement one shipped to me at no cost to me.

Nicely handled, even if the website was littered with spelling mistakes.

[23:01] [tech] [permalink]

Saturday, 13 June 2009

The Lenovo T400

My new laptop that I ordered a while ago finally arrived on Tuesday.

I wasted no time excising Windows Vista Basic that came with it and installed Debian 5.0 (Lenny)

The installation went fine, and the laptop is running well.

I'm running the 2.6.29 kernel from backports.org, because without it the WiFi doesn't work. The in-kernel HDAPS support doesn't seem to work with this kernel, and I'm having some problems building the tp-smapi kernel module, which I haven't looked into too much yet. The fingerprint reader is currently unsupported, so no biometric shenanigans just yet.

The biggest annoyance is that I didn't actually compare the resolution of the display with the resolution of my old laptop (the D610). The display is only 900 pixels high, compared with 1050, so I can't get a 2 x 2 tiling of Gnome Terminal windows at the default font size, which is somewhat irritating, as that's how I'm used to working. Maybe the solution is to start using more virtual desktops. Or use Alt+Tab more. I'm pretty sure the X301 had the same display resolution, so I'm feeling somewhat vindicated in my decision to not spend twice as much money on the X301, as that would have just made me even more annoyed with myself.

The default minimalist X.org configuration file was having X default to using the VESA driver instead of the Intel one. This was producing only a 16-bit colour depth, and resuming from suspending to RAM wouldn't turn the backlight back on. It took me a few days to figure out that the VESA driver was in use, and as soon as I swapped to the Intel driver, the backlight problems went away as well, so that was a win. I haven't bothered to try and get Compiz working yet.

Aside from that, everything's working well. The camera worked out of the box. It resumes from being suspended to RAM ridiculously quickly. The hard drive is super quiet. I've nearly finished moving out of the D610. Oh, and most importantly, it runs so much cooler than the D610.

Because this laptop is now the gruntiest machine in the house, I'm going to stop doing all my package builds on caesar and do them on this laptop instead, so I've set up pbuilder and schroot and whatnot. The CPU has the VMX extensions, so I can have a play with KVM as well.

[11:18] [tech] [permalink]

Thursday, 04 June 2009

Software controlling the power to USB Woot! lights

I've had a few enquiries from my blog post about trying to control dumb USB-powered lights.

I thought I'd just write something up to save myself replying to any more emails.

Yes, it's doable. Finding a USB hub that will do it is another story. From my own research, I found someone else who was doing something with USB-powered devices (I can't remember what now), and he had been using a what is now a Linksys ProConnect USB 4-Port Hub USB 2.0

I'd first tried a couple of random cheap hubs from Fry's with no success (fortunately I was able to return them) before I determined that the Linksys one would definitely work. The downside of the Linksys hub is it requires external power. It was also one of the more expensive USB hubs on the market.

One of my co-workers, who is an Electrical Engineer by education, said that the USB spec requires the functionality that I wanted, but most chip manufacturers had cut a corner in the interests of cost saving. The Linksys hub uses an NEC chipset. Every other hub that I could get my hands on had a Genesys Logic chipset, and did not work. You can tell if you've got a winner by the output of lsusb -v. If the hub characteristics include "Per-port power switching", you're in business.

To do the actual port powering on and off, I'm using a setuid-root hub-ctrl, wrapped with a small shell script, which has the USB ID of the hub and the port number the lights are plugged into hard-coded in it.

In my searching, I found also that it may be possible to do with Python, but I did not invest the time trying to find out.

[23:20] [tech] [permalink]

Saturday, 16 May 2009

Deciding between the Lenovo T400 and X301

I'm in the market for a new laptop. I've got almost 4 years out of my Dell D610 (which I've been despising from not long after purchasing) and decided that I'll never buy from Dell again.

I used to be very down on ThinkPads because of the placement of the Escape key, but since I've been using one at work for the last 3.5 years, I've gotten used to the Escape key location, and found the laptops themselves to otherwise be very solid and well supported by Linux.

So Lenovo it is.

Next question: T400 or X301? I've seen both, and the X301 is very sexy in terms of thickness, weight, and having all the latest bells and whistles, but boy, is it expensive.

I did a side-by-side comparison between a T400 decked out the way I wanted, and an X301 decked out the best I could choose, and I've opted for the T400.

The main problems I had with the X301 were that there was no choice of processor, and the processor available seemed significantly slower than what I could get in the T400, and of course the cost. I seemed to be paying nearly $1000 more for a slower processor, just to get something extra thin and light, with LED backlighting and an SSD disk.

What finally swayed me to spending less money and getting an arguably "better" laptop was if I paid less now, I'd be more comfortable possibly getting something in less than 4 years time down the track. So if it came down to the X301 for 4 years or the T400 for 2 years, the T400 seemed like a better deal.

What will hopefully end up happening is I'll end up using the T400 for four years and not end up hating it by the end of that time.

Here's what I ended up going with:

  • Intel® Core™ 2 Duo Processor P8600 (2.40GHz 1066MHz 3MBL2) 25W
  • 14.1 WXGA+ TFT, w/ CCFL Backlight, Camera
  • Intel Graphics Media Accelerator 4500MHD with vPro
  • 2 GB PC3-8500 DDR3 SDRAM 1067MHz SODIMM Memory (2 DIMM)
  • UltraNav (TrackPoint and TouchPad) with Fingerprint Reader
  • 160 GB Hard Disk Drive, 7200rpm
  • DVD Recordable 8x Max Dual Layer, Ultrabay Slim (Serial ATA)
  • Intel WiFi Link 5100 (AGN) with My WiFi Technology
  • 9 cell Li-Ion Battery

All of that plus a 2 year warranty and an additional 90W AC adapter came out at $1,240 (they have a 15% off discount code thing running until tomorrow, which brought the price down a bit)

I'm particularly excited about not having a proprietary graphics chipset. I'm particularly unexcited about it including Windows Vista Home Basic.

Drat, enumerating all of the options made me just realise that I seem to have omitted the 5-in-1 media reader from the final order. Oh well. That was largely just a nice-to-have anyway. Sarah tends to do all of the photo management on her laptop.

I doubt this will get to me before Thursday, so I should have it waiting for me by the time I get back from Barcelona.

[12:41] [tech] [permalink]

Monday, 04 May 2009

Analog cable's days numbered?

Apparently Comcast is planning on axing its analog cables at some undetermined point in the future.

That will make my MythTV setup more complicated, and no doubt require a more expensive cable plan and yet more equipment. My setup is already a tad ridiculous.

[18:30] [tech] [permalink]

Sunday, 12 April 2009

Queensland Rail to get WiFi

I just read that they plan to put WiFi on the trains in Brisbane.

I had one of my crazy business ideas to try and do this years ago. I think it'd be great for public transport adoption. Imagine being able to be on top of your email by the time you arrive at your desk?

Of course, with mobile broadband becoming more and more ubiquitous, you don't really need WiFi on the train, so it may be too late...

[22:32] [tech] [permalink]

Saturday, 04 April 2009

It's not just me

Neil put me onto this guy who is also doing monitoring of his cat water bowl. Unlike my way of doing it, he seems to be doing it without any probes in the water, which would make refilling the water bowl a lot less fiddly, although adds the requirement to have a rig above the bowl.

This ultrasonic sensor thing sounds very interesting.

I foresee the soldering iron making a reappearance in the near future...

[12:06] [tech] [permalink]

Asterisk Gateway Interface 1.4 and 1.6 Programming

Asterisk Gateway Interface 1.4 and 1.6 Programming

I received an email out of the blue recently from someone related to Packt Publishing, the publisher of Asterisk Gateway Interface 1.4 and 1.6 Programming, asking me if I'd like a review copy of the book.

This is the second unsolicited book review request I've received now. I just wish I had more time/mental energy to read books!

I'm going to make a concerted effort to read this one and write a review of it, because AGI is something I'm moderately interested in. AGI is what makes a lot of computer-telephony integration really possible with Asterisk, and CTI done right is something I'm particularly interested in.

I was also given some sample content and encouraged to post it. Based on this sample content, the book sounds like it should be good.

[10:47] [tech] [permalink]

Monday, 23 February 2009

Upgrading MythTV to Debian 5.0 (Lenny)

I've been dying to upgrade my MythTV setup to use Lenny for ages, ever since I got a HD HomeRun and discovered my primary MythTV front end couldn't play back the HD recordings for some reason (I just got a blue box and the audio, and a lot of errors in the X server log).

I figured it was either the kernel or X, and I hadn't been able to get the lirc kernel module to build against anything newer than Etch's 2.6.18 kernel (as I discovered today, that was because of some contamination in module-assistant's build directory, so I could have tried a newer kernel ages ago - drat), so I was unable to determine if it was the kernel or X.org itself.

An added bonus of upgrading X was it talks to the TV better. I've got the PC hooked up to the TV's VGA input, and previously, the best I could do was get it to talk 1024x768, with the TV set to a 4:3 aspect ratio. When I initially upgraded, for whatever reason, the TV was very unhappy with what it was receiving on the VGA input and wouldn't display anything, so in desperation, I tried ditching the xorg.conf entirely, and then everything just worked. X came up with a 1280x768 resolution, I think despite the TV thinking it's getting 1024x768, so now we can have the TV in 16:9 mode, and things don't look all stretched, even when the playback is only 4:3.

And HD, oh it looks glorious. We just watched the Oscars, and it completely filled the TV and looked absolutely fantastic. Playing back HD content seems to really peg the CPU though. The playback was unbearably choppy until we paused the commercial flagging job that was madly trying to flag commercials in the same recording we were currently watching (i.e. the Oscars).

The only thing that seems to be acting a bit weird is notification pop ups from the D-Bus notification daemon. I've got a Griffin PowerMate that runs a home-grown script to re-pair the Bluetooth keyboard and mouse, and that uses the D-Bus notification system to provide feedback while it's running. It seems that while the MythTV front end is running, those pop ups aren't displaying correctly. I haven't gotten to the bottom that one yet.

I also had some problems with a couple of init scripts and splashy causing bootup to hang, but I'm not sure that's completely reproducible (I don't reboot this machine all that often) so it may have been operator error.

Overall, very happy with the outcome. The upgrade was fairly smooth, and I've got the additional functionality I've been waiting for.

[00:01] [tech] [permalink]

Wednesday, 04 February 2009

PlayOn. Coming soon to the Wii, but you need Windows?

I just learned about PlayOn, which allows you to watch Netflix and Hulu content on various game consoles, with support for the Wii coming in early 2009.

This all sounded good, but it seems that it relies on a Windows PC to stream the crap to the game console, it doesn't allow the game console to do it all itself. As I don't have a Windows PC in the house (at least not one that's always running Windows, Sarah can reboot her Mac into Windows if the need arises) this is of little use to me. No sale.

[22:34] [tech] [permalink]

Saturday, 31 January 2009

Oh the irony

Content analysis details:   (5.9 points, 3.8 required)                          
                                                                                
 pts rule name              description                                         
---- ---------------------- --------------------------------------------------  
 1.0 NO_REAL_NAME           From: does not include a real name                  
 1.0 HTML_MESSAGE           BODY: HTML included in message                      
 0.0 BAYES_50               BODY: Bayesian spam probability is 40 to 60%        
                            [score: 0.4956]                                     
 1.4 HTML_10_20             BODY: Message is 10% to 20% HTML                    
 1.5 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts         
 1.1 HTML_MIME_NO_HTML_TAG  HTML-only message, but there is no HTML tag         

That's the registration verification email from the National Do Not Call Registry.

[18:42] [tech] [permalink]

Saturday, 17 January 2009

USB power switch, part 2

Continuing with my mission to control the power to some dumb USB-powered lights...

Dann Frazier confirmed my theory that a USB hub would indeed do the job. I'd already found the hub-ctrl.c program he mentioned, but couldn't get it to work with my built-in USB ports of my laptop.

It seems it all depends on whether the hub will support per-port power switching or not. (lsusb -v will tell you).

So off to the mighty institution that is Fry's Electronics I went.

The first two attempts failed, as did a borrowed hub. It seems most (at least the cheap one) use a Genesys Logic chipset, which does not support per-port power control. Fortunately I chose products that weren't in blister packs, so I was able to return them to Fry's in as-new condition and try again.

Now that I knew USB hubs would do the trick, I did some more targeted searching, and found this thread where someone had been messing around with hubs to do what sounded like what I wanted. (Incidentally, this email in the thread also provided a nice looking Python program. I'm going to look at refitting it to use the "real" Python USB module.)

I emailed the poster to ask him what brand of hub he was using. The answer was the Linksys 4-port hub

So I managed to track down one of them last night. Yes, it works, but the downside is the hub itself requires external power, which is a bit unfortunate. Not only do I need to use a hub to make these lights software controllable, I have to plug the hub into a power outlet. Bleh.

[12:21] [tech] [permalink]

Thursday, 15 January 2009

On filesystem layouts

Wouter's post prompted me to write this...

I find it particularly strange to have a single filesystem for everything, especially so on a laptop. I worked with a guy once who had the single filesystem (I think it was ext2 or ext3), and his laptop crashed, and the filesystem was trashed so badly from the crash (I remember: it was a PowerBook, so I don't know if that had anything to do with it) that it was a write-off.

One of the reasons I use multiple filesystems apply just as much to a laptop as to anywhere else: loss/damage compartmentalisation. If one of my filesystems decides to eat itself, odds are I can still get the others back.

Wherever I go, I tend to have a root filesystem, one for /usr, one for /var, one for /tmp and one for /home. If it's a server, I may have a separate one for /srv as well. I always use LVM, and with ext3 supporting online resizing these days (well online enlarging at least), it's not a big problem to leave some logical extents unallocated, and just grow whichever filesystem needs it as it needs it.

[22:29] [tech] [permalink]

Tuesday, 06 January 2009

On trying to use a Sansa e260 on Linux

I received a SanDisk Sansa e260 (v2) for my birthday. It was one of those presents that acquired for myself on behalf of someone else because it worked out easier that way. I bought it for cheap from Woot!

I thought I'd use it to play podcasts while I'm at the gym. We've been listening to NPR for a while now, and there's lots of excellent programming, and it's all available in download for as well. Since I don't spend all that much time in the car, I'm not listening to much of it.

The first mission was to upgrade the firmware. Getting the player into MSC (instead of MTP) mode with the original firmware was a little tricky, but I got there in the end. Then I had to locate the firmware in a Linux-installable format. I think I ended up resorted to using Windows, but I later found the raw firmware image without needing an installer.

I'm using Rhythmbox at the moment to download the podcasts. I thought I'd try out its MTP support, just because it actually had it, but either Linux 2.6.26 has a problem with MTP, or the player is particularly flaky doing it with Linux, because I just seemed to get a pile of USB errors when connecting the device in this mode, so I'm sticking with MSC for now.

Oh, and I had to put a .is_audio_player file in the root of the device to make it show up in Rhythmbox. I later discovered that it worked most optimally if I put

audio_folders=MUSIC/
folder_depth=2

in the file.

I'm still trying to come up with an optimal way of managing the audio I want to listen to. It seems somewhat tedious to drag and drop all of the podcasts from Rhythmbox to the device. I kind of want some sort of plug and play auto-sync method. It seems I can't use a connection-triggered rsync, because some of the podcasts seem to get downloaded with non-FAT-filesystem-friendly filenames. Rhythmbox seems to take care of this by doing some sort of automagical renaming at transfer time. That said, not everything is showing up in the music list with sane names. There's a few things that have an unknown artist I think. That may be a deficiency in the ID3 tags of the MP3s themselves though...

What I really want some way of getting home, hooking up the player and just getting what's new, without expending too much effort (at the time of hooking up the player, anyway. I'm happy to expend some effort getting to that point)

The next weird thing is that the device wants to talk USB 1.1. I'm not sure what is at fault here, but I can, allegedly at least, make the device reconnect at USB 2.0 by removing and reloading the ehci-hcd module after connecting it.

Here we have the device initially attached:

apollock@icarus:~$ lsusb -t
Bus#  5
`-Dev#   1 Vendor 0x1d6b Product 0x0002
Bus#  4
`-Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  3
`-Dev#   1 Vendor 0x1d6b Product 0x0001
  `-Dev#   9 Vendor 0x0781 Product 0x7423
Bus#  2
`-Dev#   1 Vendor 0x1d6b Product 0x0001
  `-Dev#   6 Vendor 0x413c Product 0x8103
Bus#  1
`-Dev#   1 Vendor 0x1d6b Product 0x0001

lsusb says that bus #3 is ID 1d6b:0001 Linux Foundation 1.1 root hub

Then if I do a spot of presto-changeo...

apollock@icarus:~$ sudo modprobe -r ehci-hcd
apollock@icarus:~$ sudo modprobe ehci-hcd
apollock@icarus:~$ lsusb -t
Bus#  5
`-Dev#   1 Vendor 0x1d6b Product 0x0002
  `-Dev#   3 Vendor 0x0781 Product 0x7423
Bus#  4
`-Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  3
`-Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  2
`-Dev#   1 Vendor 0x1d6b Product 0x0001
  `-Dev#   7 Vendor 0x413c Product 0x8103
Bus#  1
`-Dev#   1 Vendor 0x1d6b Product 0x0001

Bam! It's moved over to bus #5, which lsusb says is 1d6b:0002 Linux Foundation 2.0 root hub

Weird.

I wonder if I should try running Rockbox on it? Seems not, it's only for v1 players. Oh well.

[22:48] [tech] [permalink]

Sunday, 04 January 2009

Feed breakage, Blosxom's shiteful RSS 0.91 implementation

(or echo "blosxom hold" | sudo dpkg --set-selections)

A few people had pointed out to me that my blog was appearing weirdly in Google Reader. I myself read my blog via Planet Debian and Planet Linux Australia, where it appears fine, so I hadn't noticed the problem first hand.

I did some digging tonight with the Feed Validator, and I think I've figured it out.

Funnily enough, it all started with a previous session with the Feed Validator...

I wanted my blog to be UTF-8, with it being the new hotness and all. Blosxom was spitting it out as ISO-8859-1. I've just learned from this evening's debugging, that this is the behaviour of CGI.pm's header() function if not provided with the -charset parameter.

I'd previously worked around this by defining my own content_type for the RSS and HTML flavours. In this file, I just put text/xml; charset=utf-8and text/html; charset=utf-8, respectively.

Unfortunately, this then breaks Blosxom's internal HTML-escaping-for-RSS-feed code, which is conditional on the content type being XML:

if ($content_type =~ m{\Wxml$}) {
        # Escape <, >, and &amp;, and to produce valid RSS
        my %escape = ('<'=>'&lt;', '>'=>'&gt;', '&'=&gt;'&', '"'=>'&quot;');  
        my $escape_re  = join '|' => keys %escape;
        $title =~ s/($escape_re)/$escape{$1}/g;
        $body =~ s/($escape_re)/$escape{$1}/g;
      }

This block of code was being ignored because the regex was anchored. "text/xml; charset=utf-8" didn't match "\Wxml$". Whenever I last had an attempt to make my blog's feed pass the Feed Validator, I must have edited the Blosxom source code to make this regex not be anchored.

Then on November 11, 2008, I updated my Blosxom package to 2.0-14+etch1, to address CVE-2008-2236 (incidentally, there doesn't appear to be a Debian Security Advisory for this?), and my local modification got clobbered. That'll teach me to make local modifications and not put the package on hold. Blah.

It seems I'm screwed, there's no way to have a well-formed UTF-8 RSS feed without requiring a source-code modification to Blosxom. I've either got to change the way the header() function is being called, or I've got to change the regex to allow me to specify a charset in the flavour file, and not disable HTML escaping.

Speaking of screwed, Blosxom seems incapable of making a well-formed RSS 0.91 feed anyway. It wants to put the <pubDate> tag inside an <item>, which is a violation of the DTD. There's supposed to be one for the whole <channel>, not one per item (or "story" in Blosxom-flavour-speak). You can't just move that from the story flavour file to the head flavour file, because a pile of variables that you'd otherwise use seem to only get set on a per-story basis. Double blah.

I'm starting to feel like Blosxom is outliving its usefulness, but I really do not feel like embarking on a wholesale move to a new blogging platform, I'm fairly entrenched.

[22:35] [tech] [permalink]

Mixing electricity and water: monitoring the cat water bowl with Nagios

(this is something I've had "in production" for many months now, I just haven't had the time or energy to do a proper write up about what I did)

We have a cat water bowl, it looks like this:

The cat water bowl

Under "normal" circumstances, it usually lasts about seven days. So when our weekly routine is happening, we'll refill it on a Saturday whilst doing house chores, and it'll last until the following Saturday when it gets refilled.

Unfortunately, sometimes our routine gets disrupted, and we forget. Sometimes, we travel and have a house-sitter, who may not pay as close attention to such things as ourselves.

Once, one of our cats was licking the condensation off a chilled bottle of soft drink that was on the kitchen counter one evening, before we realised the water bowl needed refilling. Naturally, we felt like terrible pet owners.

So I think it was some time around the 2008 Maker Faire, that I hatched the idea of having some sort of water sensor on the cat bowl, which would communicate to one of the various computers in house. At the Maker Faire, I bought a copy of Making things talk, and an Arduino starter kit, which consisted of a Diecimila board and a make-it-yourself proto shield. I also bought a little electronics starter kit, which consisted of a breadboard and various components, and a USB-TTL cable.

I decided to use Bluetooth to communicate with the board, as I already had my MythTV setup using a Bluetooth keyboard and mouse, and it was within range of the water bowl. I decided against using Zigbee, because I didn't know anything about it, and I didn't want to add (or learn about) yet another wireless infrastructure just for this project.

I should point out that I know very little about electronics. I'd never owned (or really used) a soldering iron until I embarked on this project. I took a basic soldering class at the Tech Shop, but I'd already assembled the proto shield by the time I took the class, so I'd pretty much figured it out.

I had a very naive vision that I could just basically shove two wires in some water and it'd close a circuit and that would be my water sensor. Of course this didn't work, so I started hunting around on the Internet for a circuit that would do this. I happened upon a circuit (I don't seem to have retained the URL, so I can't link to it), which just consisted of a couple of transistors and some resistors. So I headed off to Fry's to try and buy the transistors I needed.

I quickly discovered that I didn't have sufficient information to identify the transistors that I wanted, but I did happen to stumble upon a cheap assemble-it-yourself water alarm. It consisted of a PCB, and a transistor and some resistors and a buzzer. I bought a couple of these instead.

Between studying the PCB and the circuit diagram that came with the alarm, I was able to reproduce it on my breadboard instead of on the PCB. Sure enough, placing the two probes in water closed the circuit. I replaced the buzzer with an LED so I could see what was going on.

I attached the circuit to the Arduino proto shield, and had it feed into one of the digital I/O ports. I wrote some quick and dirty Wiring code so that when water not present (i.e. the circuit was open and no current was detected on that I/O port) the LED was switched on. Really at this point, I didn't need a microcontroller, I could have presumably achieved the same thing with a NOT gate.

At this point, I wanted to make the sensor remotely queryable. I bought a BlueSMiRF Silver Bluetooth modem, which I attached to the TX and RX lines of the board (I first configured it by attaching the USB-TTL cable to it and using Minicom on my laptop). I extended the Wiring code to provide a rudimentary prompt, and accept a command to check if water was present.

I think around this time it also dawned on me that I could use the digital I/O pins as a switch. When they're "high" they provide power. So rather than constantly running a current through the water, I only needed to briefly power up the water detection circuit, see if the circuit closed or not, and then report if water was present if it did.

I much preferred this, as at the time, I was endeavouring to power the whole sensor off a 9 volt battery. I figured I'd get much better battery life if I wasn't running a current through the water the entire time. I should point out that I did some "tongue tests" in a glass of water while the circuit was powered up, and couldn't detect a difference between when the circuit was on or off. The last thing I wanted to be doing was zapping the cats!

At this point, the Wiring and Arduino work was pretty much complete. I setup ser2net on the MythTV server, so that I could just connect to port 4000, and be connected to the water sensor.

apollock@icarus:~$ telnet teevee 4000
Trying 172.16.0.9...
Connected to teevee.andrew.net.au.
Escape character is '^]'.

ser2net port 4000 device /dev/rfcomm0 [9600 N81] (Debian GNU/Linux)


waterbowl> s
Water is present
waterbowl>
telnet> q
Connection closed.
apollock@icarus:~$

The Wiring code running on the Arduino board is checked in here.

Next, I wanted to monitor this with Nagios. One thing I found with the Bluetooth connection was that it wasn't all that reliable. Not every connection to port 4000 resulted in a connection with the water sensor. I elected to write some standalone code that submitted results to Nagios by way of a passive check, rather than having Nagios try to actively monitor it. Again, trying to conserve energy, I decided to only check the sensor once every 8 hours.

So I wrote a Python daemon, which tried to be as robust as possible. If at first it doesn't make a connection on its 8 hourly schedule, it keeps trying until it does. Nagios itself has some freshness detection for this monitor, so if no passive result is submitted within 8 hours and one minute, Nagios alerts that something is possibly wrong with the sensor itself. (The original intent was to deal with the situation where the battery went flat)

This is the Nagios service definition I've got. Some of it may be unnecessary or redundant:

define service {
        host_name                       teevee
        service_description             Cat water bowl
        check_command                   check_stale!2!"Check water bowl monitor is on and reachable"
        normal_check_interval           480
        notification_interval           240
        active_checks_enabled           0
        check_freshness                 1
        freshness_threshold             28860
        max_check_attempts              1
        check_period                    24x7
        use                             generic-service
        stalking_options                o,w,c
        contact_groups                  everyone
}

The code for the Python daemon is checked in here.

So everything was going swimmingly, apart from I couldn't get even 24 hours of monitoring on my conservative 3 times a day schedule, and the Bluetooth modem configured for all of its power saving options, and the water sensor itself only being powered on when it was performing a check. There wasn't much more I could do to try and reduce power consumption, at least with my limited electronics knowledge. Perhaps using Zigbee instead of Bluetooth would have helped, but I still think I'd have been going through 9 volt batteries more quickly than I'd have liked.

I also had a brief foray into solar powering the board. I thought maybe the lighting in the house would be sufficient to power the board. Of course this didn't work out. I'd also have needed to make the code running on the board more self-sufficient, and have it just provide a water status indication whenever it had enough power to do so, instead of being a fairly dumb device like it currently is. This all felt a whole level more complicated, and out of my league, and I wasn't interested in attempting this sort of remote-sensing exercise at this time.

The Arduino board can also be powered by USB, so as I already had some long USB type A to type B cables (that had funky lights in them to boot), I decided to just get a wall wart that had a USB type A receptacle, and powered the Arduino board that way. So much for being completely "wireless". (I could have also gotten a general purpose DC power supply that was capable of spitting out 9 volts, but I doubted I'd get one with a sufficiently long cable, which is why I went for the USB-powered option, as I already had a cable long enough)

Speaking of wires, the most challenging part was the probes for the water sensor. As they were going to be permanently in the cats' drinking water, I didn't want to contaminate the water with them. I figured plain untinned copper wire would be okay, since water pipes are copper. Finding untinned, unstranded copper wire was a real challenge. I started out using some FM antenna cable, but that was stranded, and the shielding was a nightmare to strip. It was also reasonably difficult to make go where I wanted it.

What I really wanted was something more solid, that would flex and stay where I put it. I cannibalised a spare IEC power cable, but it was also stranded copper wire. I finally managed to obtain some solid-core CAT-5 cable from a hardware store, and this has worked exactly as I wanted.

I haven't done any further work on the setup since getting the CAT-5 cable for the probe. Further improvements that I'd like to do at some point:

  • transfer the circuitry from the breadboard to a breadboardless proto shield, and generally try to house the sensor better
  • find a better way of connecting the probe wires to the circuit, so it's easier to remove the monitor when refilling the water tank (currently the CAT-5 wires are just shoved into the breadboard). The whole thing is barely spouse-approved as it currently is, let alone being easy enough for an untrained house-sitter to try and negotiate
  • integrate with Asterisk so a house-sitter gets a telephone call when the water bowl needs refilling

The finished product

I had a lot of fun with this project. I had a real sense of achievement, being able to go from concept to completion, and learn a few things about electronics along the way. I'm normally not a fan of messing around with hardware.

Some photos of the project are here.

[16:55] [tech] [permalink]

Thursday, 01 January 2009

USB power switch?

I bought these USB powered lights from Woot! recently, with grand plans of using them with a continuous build/test/install rig to provide visual notification of when a build/test/install completed or failed or something.

Somewhat to be expected for the price, they merely draw power from the USB port, and have a physical switch on each light. I want something software controlled though.

There's something in /sys/bus/usb/devices/*/power/level that looked promising, but seems to be a red-herring (at least for non-devices. I was able to turn off a USB mouse and a Bluetooth adapter this way)

What I'd like (and it probably exists, I just have to find it), is a USB device that presents itself to the operating system, and its sole purpose is providing pass-through power to a USB A-type receptacle on itself. This would then be manageable by the operating system.

What I'm possibly trying to say here is a small, specialised, one port USB hub. Hmm... I'll have to see if I can borrow a USB hub from someone to test this theory. I've never had a need to own one.

[17:44] [tech] [permalink]

Wednesday, 31 December 2008

On unlocking a Vonage-provided Linksys RTP300

Well that was a bit of process.

The best set of instructions I found were here.

I think a lot of the other instructions I found assumed you were at firmware version 1.00.62. The CYT unlocker program seems to only work with this firmware version.

The CYT unlocker program is a Windows console-mode program. It seems to run under Wine, but I switched to Windows before I realised that possibly my problems running it under Wine were related to the firmware version on the RTP300 and not a problem talking to the network.

All of the remaining steps on the above page could have been performed on Linux. In fact, the CYT unlocker program merely makes a HTTP POST to a specific URL (and I think it could just as easily be an HTTP GET) and then starts up a webserver on port 2400 to hand an XML blob of provisioning information when the RTP300 requests it as a result of the HTTP POST. A Linux version would be easy to implement.

In all of the futzing around, I noticed this gem in the HTML source of the web interface:

*********************************************************
*   Copyright 2005, CyberTAN  Inc.  All Rights Reserved *
*********************************************************

This is UNPUBLISHED PROPRIETARY SOURCE CODE of CyberTAN Inc.
the contents of this file may not be disclosed to third parties,
copied or duplicated in any form without the prior written
permission of CyberTAN Inc.

This software should be used as a reference only, and it not
intended for production use!

Now that I've unlocked it, I guess I should try and make it talk to my Asterisk box...

[15:43] [tech] [permalink]

Goodbye Vonage, hello T-Mobile@Home

When we first moved to the US, I wanted to try and do the telecoms and Internet stuff as cheaply as possible, while still trying to be technologically "fun" as well.

I elected to get the cheapest home phone line (local calls only) I could get with AT&T (then SBC) (which is still remarkably expensive) so that I could get DSL (which, while three times faster than what I'd had previously in Australia, was still not that fast). (I went for DSL over cable because I'd heard horror stories about Comcast's reliability, and also they weren't cool with running servers or inbound services, Sonic is, and is a great ISP). I chose Vonage for US long-distance, I think because I'd heard of them previously, and was interested in trying out VoIP.

We initially used Vonage for calling Australia as well, until I played around with Engin, and then deployed Asterisk.

Anyway, I've been a loyal Vonage customer for exactly three years. The truth is, we hardly make any US long distance calls. We've got friends in Phoenix (hi Craig and Sarah!) that we call infrequently. I guess there might be the odd cell phone that isn't considered a local call, but we tend to use our cell phones more than the home phone anyway... So we were a very cheap $14.99 a month for Vonage.

So when I called T-Mobile to purchase a SIM card for the Android Dev Phone 1 I'd been recently given, the opportunity to have their T-Mobile @Home in place of Vonage, for an extra $9.99 on top of what I was paying for the cell phone line seemed like worthwhile cost saving. I have no attachment to the incoming number for the Vonage line, we give out the AT&T land line number to people as our "primary" home number.

Later that day, Vonage announced they were increasing their monthly fee from $14.99 to $17.99, which made the decision seem all the more prodigious.

I did a spot of research while I was waiting for the @Home box to ship to me, and it turns out that they have a wireless router and non-wireless router option. They hadn't specified what I was getting at order time, and I was hoping I'd get the non-wireless router, since I already had an access point I was perfectly happy with it. As luck would have it, I received the wireless router model (a Linksys WRTU54G-TM). I'd mentally prepared myself to be receiving just another Linksys RTP300, like what Vonage used, and so I was quite surprised by the differences in technology used.

It's been a while since I tried to do any reverse-engineering of the Vonage ATA, but I think from memory, it was essentially that: an ATA. It talked SIP back to Vonage. My DSL provider gave me a few static IP addresses, so when I got it, I just allocated one for Vonage, and did a static NAT through my firewall to for it, and everything just worked.

It was fairly apparent that the WRTU54G-TM was nothing at all like this (given it was also an access point). It takes a GSM SIM card (up to two in fact), which I presume has all of the provisioning information on it. I was initially worried that I was going to need to completely rejig my network to accommodate it. I guess the SIM card approach means there's no customer specific provisioning that presumably otherwise needs to be done to the device itself, and they can just ship vanilla devices. I presume the Vonage RTP300 had some sort of customer-specific configuration, because I never had to do anything.

I briefly toyed with the idea of running two access points, and having a "guest" wireless network, but I am interested in reducing heat and power consumption in my linen cupboard, so I decided to try and just swap over to using the WRTU54G-TM as my access point, retiring my Linksys WAP54G.

So that's exactly what I sat down to do last night. Sometimes in their efforts to make these things "simple", they end up making things more complicated. What I wanted to do was just plug the "Internet" side of the device into my internal network, have it get an IP address with DHCP, and then let me get on with configuring it. The "advanced" setup instructions talked about the box having a default IP address of 192.168.0.1 ("or try 192.168.24.1 if that doesn't work"). I figured the DHCP IP address would override that, so I was trying to hit the IP address I could see the unit had picked up from my DHCP server.

No response from the HTTP server. I could see it replying to ARP requests, and I could see what appeared to be an IPsec NAT-T connection. After some fiddling around, I twigged that I was trying to manage it over the Internet port, and any device worth its salt was going to be very hardened out of the box on what it expected to be the raw Internet-facing side, so I plugged my laptop directly into one of the four Ethernet ports that it also had, and then I could hit the 192.168.0.1 IP address and access the management interface.

It never occurred to me prior to fighting with getting the WRTU54G-TM configured, that the RTP300 might have been more than a black box, as I'd never tried plugging anything into the "inside" ports on it, mainly because I'd never intended to use it as a router or a switch.

Out of the box, the WRTU54G-TM was configured to be a router, which I didn't need, and it was configured to be a DHCP server for 192.168.0.0/24, which I definitely didn't want. An interesting side-effect of putting it into bridging mode was the MAC address changed (to the one that was printed on the unit, which was not how out of the box it presented itself). I have no need or intention of plugging anything into the "inside" Ethernet ports on the WRTU54G-TM, as I have a separate switch for that, and indeed I run my wireless network on a different interface of my firewall to my wired network. Fortunately I was able to enable the management interface via the "Internet" interface.

The added benefit of the device doing NAT-T is I don't need to allocate an IP address to it for the VoIP stuff it work, it all just works fine without it, so I've freed up an IP address, removed two devices from my linen cupboard, added one new one, and saved some money. Woot!

Next, just because I can, I'm going to try and break into the RTP300 and see if I can reconfigure it as a general purpose ATA to talk to Asterisk. There's evidence it's possible. It's otherwise of no further use to me. Even if I get it working with Asterisk, it's not of a lot of use to me. Maybe I can send it to a family member, and they can call us over the Internet for free instead of for the cost of a call to Brisbane...

[12:30] [tech] [permalink]

Saturday, 20 December 2008

The Bad Block HOWTO hurts my brain

Seriously.

It reads more like a worked example, without enough of the underlying theory. It doesn't help that I have a slightly more complicated situation, in that I have a filesystem on a logical volume that spans 6 physical volumes.

So maybe writing all of this down will help...

I know the disk with the problem: /dev/sdc

I know the LBA of the sector where the rot starts: 754238963

I know that the physical volume is on /dev/sdc1 and that the partition starts at sector 63 (via sfdisk -luS /dev/sdc)

Therefore, within /dev/sdc1, we're talking about block number 754238899 (754238963 - 64)

I know the physical extent size is 4096 Kb (via pvdisplay -c /dev/sdc1 | awk -F: '{ print $8 }'). In LBA block size this becomes 8192 (2 x 4096)

I know the physical extents start 192K (into the partition, presumably) (via pvs -o+pe_start /dev/sdc1)

This is where the Bad block HOWTO starts to quicken the pace a bit on me...

I'm supposed to take the physical partition's bad block number (754238899) and divide it by the size of the physical extent (in LBA block size) (8192). So that'd be 92070.

This is where I think I end up completely departing the HOWTO, because my filesystem spans multiple physical volumes. Here's my general musings...

lvdisplay --maps /dev/data/srv tells me this about /dev/sdc1:

  Logical extent 272326 to 391559:
    Type                linear
    Physical volume     /dev/sdc1
    Physical extents    0 to 119233

So I'm inferring from this, given that I know I want physical extent 92070 of this device, that this corresponds to logical extent 92103 (again, I'm expressing this in LBA block size, so that's (272326 / 8192) + 92070

So now I supposedly know the logical extent (in LBA size) of the bad block. Now what? I think I'm wandering up too many layers, closer to the physical filesystem, but I'll continue wandering around...

So presumably at this point, I just want to convert the logical extent to an actual filesystem block number. Maybe the logical extent number divided by the extent filesystem block size divided by the LBA block size? 92070 / (4096 / 512) = 11508. That feels awfully low though. The dd test suggests this is incorrect.

At this point I'm feeling fairly lost. If someone out there is reading this, and they've done this before, I'd love to hear from you.

[23:08] [tech] [permalink]

Reflections whilst waiting for fsck, part 2

So it turns out if you wait long enough, you do get to piece the information together:

.
.
.
Too many illegal blocks in inode 209158151.
Clear inode? yes

Restarting e2fsck from the beginning...
/srv contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'denemo_0.7.7-3.1_hppa.deb' in /debian/pool/main/d/denemo (211845291) has deleted/unused inode 209158151.  Clear? yes
.
.
.

[16:59] [tech] [permalink]

Reflections whilst waiting for fsck

I'm watching paint dry while e2fsck does its thing on a ~2TB filesystem (the one with all the good stuff on it on mirror.linux.org.au).

I'd seen a spate of kernel errors during the week about "attempt to access beyond end of device" so I figured it was due for one.

Let's take this output for example:

apollock@disco:~$ sudo e2fsck -y /dev/data/srv
e2fsck 1.40-WIP (14-Nov-2006)
/srv contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 209158151 has illegal block(s).  Clear? yes

Illegal block #12 (1620503259) in inode 209158151.  CLEARED.
Illegal block #13 (2992116621) in inode 209158151.  CLEARED.
Illegal block #14 (1657577172) in inode 209158151.  CLEARED.
Illegal block #15 (1168774619) in inode 209158151.  CLEARED.
Illegal block #16 (993415032) in inode 209158151.  CLEARED.
Illegal block #18 (1611893880) in inode 209158151.  CLEARED.
Illegal block #20 (2939071693) in inode 209158151.  CLEARED.
Illegal block #21 (1714919190) in inode 209158151.  CLEARED.
Illegal block #22 (1450852455) in inode 209158151.  CLEARED.
Illegal block #23 (3482149179) in inode 209158151.  CLEARED.
Illegal block #24 (4143923374) in inode 209158151.  CLEARED.
Too many illegal blocks in inode 209158151.
Clear inode? yes

Restarting e2fsck from the beginning...
/srv contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
.
.
.

How much more work would it be to tell the administrator the name of the file associated with inode 209158151? That'd be a lot more useful to most mere mortals than the inode number. I suppose if the filesystem is in a really bad state, ascertaining that information may be difficult...

Time to play with debugfs while I continue watching paint dry...

[16:12] [tech] [permalink]

On efficient shell scripting

Mike Hommey took some inefficient shell scripting to task.

I feel obligated to point out that more $file | wc | awk '{print $1}' is not the same as wc -l $file.

To get the same output, you're going to need one of:

cat $file | wc -l

or

wc -l $file | awk '{print $1}'

I suspect cat is going to have a faster startup time than awk... (On my system, gawk is an order of magnitude bigger than cat)

[13:51] [tech] [permalink]

Sunday, 14 December 2008

Asterisk call notification

My home Asterisk configuration has been pretty static for a while. It works well, and so I haven't had a need to mess around with it.

There's a couple of outstanding features I've wanted to implement for a while though, and I've now got one of them out of the way.

All calls that come in from the local PSTN line, the Vonage ATA, and the SIP connection to Engin, hit a menu, where the caller selects whether they want to speak to Sarah or myself. The sole purpose of this is to determine what to do with the call if no one answers (which voicemail box or cell phone to send the call to). A convenient side-effect of this is that it defeats most telemarketers.

Ages ago, I set up caller ID announcement to my MythTV setup, so if we happened to be watching TV or a recording when a call came in, it briefly popped up on the TV.

Fortunately, we're not huge square-eyes, and spend more time goofing off on our laptops, so I wanted a way to display notifications on our laptops when calls came in as well.

I had this crazy design in my head of having a server process on my Asterisk box, and then clients on our laptops could connect to it, and be informed of new calls. Before I went on a coding frenzy, and did something with the Observer pattern and more socket programming than I really wanted to, I thought I'd check if this was solved problem. Turns out, it kind of is.

I found the Notify application for Asterisk (which I've made a package for Debian Etch), which blats a UDP message to particular host as part of a dialplan. It's more intended to go in person-specific extensions, but as the majority of calls go into the main menu and never come out before hitting a person-specific extension, and I still wanted to know about them for my own perverse pleasure, I added multiple calls to the Notify application in the main menu dialplan.

The site for the application also mentioned some third-party clients for Linux and MacOS, but I've been looking for an excuse to try and write something that used Growl, so I wrote my own Python application, initially on Linux, and then I essentially swapped out the pynotify code for the equivalent Growl code, and it worked just as well on Sarah's Mac.

So it's not quite the solution that I had in mind, because it doesn't really scale, but for our two-user home setup, it's adequate. I might still try and write some middleware between a client for this application and an infinite number of observers, but there's no urgency.

The dialplan for our main menu now looks something like this:

exten => s,1,Background(for)
exten => s,n,AGI(mythtv-cid.agi)
exten => s,n,Notify(${CALLERID(num)}|${CALLERID(name)}|${EXTEN}/icarus)
exten => s,n,Notify(${CALLERID(num)}|${CALLERID(name)}|${EXTEN}/iapyx)
exten => s,n,Set(TIMEOUT(digit)=5)      ; Set Digit Timeout to 5 seconds
exten => s,n,Set(TIMEOUT(response)=10)  ; Set Response Timeout to 10 seconds
exten => s,n,Background(names/sarah)
exten => s,n,Background(press-1)
exten => s,n,Background(for)
exten => s,n,Background(names/andrew)
exten => s,n,Background(press-2)
exten => s,n,WaitExten

A call comes in, hits the menu, and the caller ID info is displayed on MythTV (if something's being watched) and our two laptops (if they're awake).

Unfortunately my Git-fu is atrocious, so I can't figure out how to link to a web-browseable repository that contains the source for my client, but if I figure that out, I'll update this with a link later.

Update

I managed to summon enough Git-fu to put the repository online. It's available at http://git.andrew.net.au/?p=asterisk-notify-client.git;a=summary

Now I just have to use it properly.

[23:04] [tech] [permalink]

Thursday, 11 December 2008

My SSH Tip Of The Day

There seems to be a bit of a meme here...

I have a set of computers that I frequently reinstall for testing purposes. The "zomg! The host key has changed!" gets very tiresome very quickly.

Host reinstallalot
        UserKnownHostsFile /dev/null
        StrictHostKeyChecking no

[08:28] [tech] [permalink]

Monday, 08 December 2008

RHEL doesn't have sysfsutils?

I'm reading the Red Hat I/O Performance Tuning Guide, because hey, it's all Linux, and I might just learn something.

So far, I've learned that RHEL doesn't have sysfsutils. How very weird.

Unlike the /proc/sys/ file system, the /sys/ file system does not have a utility similar to sysctl that can make such changes persistent thoughout (sic) system reboots.

and

However, as mentioned in Section 4, “Selecting a Scheduler”, any tuning made though echo commands to the /sys/ file system is not persistent throughout system reboots. As such, to make any scheduler selection/request queue settings persistent throughout system reboots, use /etc/rc.d/rc.local.

Of course, for all I know, this information may be out of date with respect to the current release, but Debian's had sysfsutils since November 2003.

[07:15] [tech] [permalink]

Monday, 01 December 2008

Rovio

I just learned about Rovio courtesy of an advertisement in last month's Wired, magazine.

My initial thought was that it might be useful for Sarah's feral cat trapping antics, but alas the FAQ says it's not recommended for outdoor use, so I doubt it would perform well in low light.

Having just watched the promotional videos, I see where this would come in very handy in the smart home I'd like to build one day... It'd be so cool if when we were away from home and the home detected an intrusion, notified me, and then dispatched a Rovio to show me what was going on. Then I could call the police if necessary.

The one immediate downside I can see is that it only seems to support WEP. How 2001.

[07:50] [tech/gadgets] [permalink]

Friday, 28 November 2008

Wising up about SMART

I've always installed the smartmontools package on my servers because I figured it'd maybe give me some early warning about an impending disk failure.

Getting proper SMART access was one of the motivating factors in moving my home-brew SAN from IDE-to-USB disks to native SATA.

Since then, I've had two drives fail on me. One of the original four that I bought, and the replacement one that Seagate shipped to me. (Incidentally, I love how your only recourse to a replacement drive failing with Seagate is they'll ship you another (reconditioned) replacement drive. I can see the shipping expenses racking up very quickly).

I decided to do the expedited return when the second drive failed, just because it meant I didn't have to be without a disk in my RAID-10 for a week while I waited for them to receive my drive and turn around and ship me a replacement drive, and it also meant I didn't have to mess around with packaging it myself.

(Incidentally, I wonder what happens in Seagate's reconditioning process? Will someone else get my failed drive (i.e. the same serial number) after they replace the platters or whatever it was that failed? Would it be interesting to set up a website where people could register the serial numbers of drives they returned to Seagate, and how they died, and at how many power-on hours? How many times does a drive get "reborn"?)

Whilst I was waiting for the replacement drive to arrive, I messed around with the failed one (I removed it from the RAID array), fiddling with the SMART self-tests that smartctl can initiate for you. The faulty drive failed the long self-test at the same LBA every time I ran it.

It was around this time that I realised that the default configuration that comes with the Debian smartmontools package isn't all that useful with SATA disks.

It ships with

DEVICESCAN -m root -M exec /usr/share/smartmontools/smartd-runner

which, as it turns out, doesn't detect SATA drives properly. I got

Nov 24 22:00:13 minotaur smartd[2990]: Device /dev/sda: ATA disk detected behind SAT layer 
Nov 24 22:00:13 minotaur smartd[2990]:   Try adding '-d sat' to the device line in the smartd.conf file. 
Nov 24 22:00:13 minotaur smartd[2990]:   For example: '/dev/sda -a -d sat' 

logged. It seems to me that if the software can detect this situation, it should be able to just try the "-d sat" behaviour automatically.

So it seems that this whole time, I haven't been getting any SMART monitoring of my SATA drives. hddtemp has been happily logging the temperatures for me though.

So I took the advice in the comments of /etc/smartd.conf and removed the DEVICESCAN entry in favour of explicitly naming the disks I wanted to monitor.

I also discovered that I can have smartd kick off scheduled short and long self-tests on whatever schedule floats my boat.

So now I've got something like this:

/dev/sda -d sat \
         -M exec /usr/share/smartmontools/smartd-runner \
         -s (L/../.././01|S/../.././(00|12))

This runs a long self-test at 1am every day, and a short one at midnight and midday. Hopefully this will help predict the next drive failure a bit earlier.

I'm now debating whether to set up an active Nagios monitor for SMART, or to see if I can write a passive one that runs as part of the smartd-runner infrastructure that the smartmontools package provides (and is pretty neat). I guess it's already setup to email me, so it doesn't really matter whether it's Nagios emailing me, or smartd itself.

[23:05] [tech] [permalink]

Sunday, 09 November 2008

Keeping hard drives cool

I've had this crazy ATA over Ethernet SAN for my MythTV setup for as long as I've been running MythTV. I'm using minotaur, an old Pentium III 1RU VA Linux server as the "head".

Originally, I had 4 IDE hard drives, attached to minotaur via IDE-USB adapters. This worked perfectly fine, but I missed the ability to query SMART information, and do other low-level drive tweaking with hddparm

So a while ago, NewEgg had a sale on 750Gb SATA drives, and I bought a 4 port eSATA card, and took the plunge with SATA. That all went fine.

The entire time, I've had the 4 hard drives themselves just sitting on top of minotaur, on cork trivets. The whole arrangement is in the bottom of our linen closet, as this is where the patch panel for the apartment is.

I've graphed the temperature of linen closet for some time now, but the benefit of having hard drives connected via SATA was that I could easily graph the temperature of them as well, which I started doing recently.

The catalyst for wanting to do this was that I had a drive fail, and I was concerned that they were getting too hot, just sitting in the closet with no airflow. Matt Bottrell's recent reminder of Google's research into hard drive failures also helped bring this concern to forefront.

Seagate's product documentation for the Barracuda 7200.10 claims that ambient operating temperature must be between 0°C and 60°C. I'm assuming that's the environment the hard drive is in, and not the temperature of the drive itself as reported by something like hddtemp. The closet is typically in the high 30's. If you believe what SMART reports, it seems to imply that anything over 45°C is a Bad Thingtm.

So after I had a drive fail, I figured I should probably try to do something about the temperature situation, and started looking around for an external enclosure.

I'd previously had problems with the IEC-to-Molex power supplies I was using to power the disks, and think that they'd been the cause of some failures of my IDE disks, so was quite eager to stop using them as well, which was something else an external enclosure would give me.

I had trouble finding an enclosure that presented 4 separate eSATA ports, and I was talking to Marc about SATA port multipliers, when he mentioned he had a brand-new unused full-height SCSI enclosure, for four disks, which was surplus to his requirements. Since all I really needed was external power and cooling, I thought I could cannibalise this to my needs.

Sure enough, I could just remove the SCSI cables and their Centronics connectors, and feed the eSATA cables out the gap they left, and use the molex-SATA power splitters I already had. Because the enclosure was intended for full-height 3.5" hard drives, there was plenty of air space between each disk. The enclosure included two fans.

The results? Well the graph speaks for itself:

Graph of hard drive temperatures

(I'm not entirely sure why minotaur's internal disk occasionally reports a very low temperature)

The added bonus is I've cleaned up the rats nest of cables in the linen closet. So thanks very much, Marc!

[14:22] [tech] [permalink]