Avoiding the Chasm

April 17, 2008

Broadband Network Usage Monitoring and Measurement Tools

Filed under: Technology — Tags: , , , , , , , , , , — vextasy @ 8:10 pm

If you have found that your ISP has been restricting your broadband bandwidth the obvious question you will ask your ISP is why? If you ask that question the answer you are likely to get is that you have been using the “broadband service inappropriately”. You might also be told that you have exceeded your usage allowance but if you, like me, are on an unlimited contract you are unlikely to be told what the upper usage limit is. The reason you won’t be told the actual value of the upper limit is that it is likely to be more complicated than a simple figure and the reason that this is the case is because the ISPs are principally interested in avoiding router congestion at peak times. At off-peak times, such as in the early hours of the morning and during the working day, it makes little difference to an ISP if the capacity of their network is 25%, 50% or 75% used, they still have the same equipment costs and other overheads. But once the network reaches capacity, and routers are forced to drop packets, then customers start to notice and the ISP begins to get a bad name. For this reason most ISPs have, quietly, begun to apply traffic shaping at peak times.

Traffic shaping involves restricting the bandwidth of `heavy usage’ customers in such a way as to prevent them from interfering with the network experience enjoyed by `lighter usage’ customers. Unfortunately it looks like we are in for a lot more of this as network demand grows. The popularity of the Internet as a medium for watching media has rocketed in recent months and looks to continue to grow as more and more people switch their viewing habits from the more traditional broadcast medium to Internet based technologies. In the UK, the BBC iPlayer alone has been responsible for tremendous changes in network usage.

I recently found myself in the position of having determined from my ISP that I was being traffic shaped. Unfortunately, my router provided me with little help in identifying the volumes of data that were passing through my broadband connection each month but my ISP furnished me with figures which seemed to be considerably higher than I would have expected. Finding myself in a very weak position I decided to rearrange my home network to allow me to gain a better understanding of my broadband usage.

My broadband is supplied by BT on an unlimited tariff and I use the BT supplied broadband router (2-Wire 1800) which hosts two wired Ethernet connections, one to a PC and one to a network attached storage device, and a number of wireless connections to PCs or laptops. There are nine devices in total but typically a maximum of 4 might be actively using the broadband line at any one time, and by active I usually mean browsing web pages. Living reasonably close to our exchange we can manage to achieve a download speed of about 6.3 Mb/sec and even with the bandwidth restriction we still reach this speed outside of peak times.



The problem with having a wired network is that the only device that can really determine how much traffic is flowing is the broadband router as everything else talks directly to it and it funnels data into and out of the ADSL line. I attempted to get my existing router to log traffic information to a PC so that I could take a close look at what was travelling up and down the external line.

There are many different logging analysers available but the one I chose to use was WallWatcher a free tool with support for a large number of routers.


Unfortunately I found that I couldn’t get WallWatcher to correctly recognise the format of the packet logs coming from my model of 2-Wire router. Linksys_WRT54GLIn my case the solution was to make use of a spare wireless router that I had which was not being usefully employed, a Linksys WRT54GL. This variant of the WRT54 family of router runs Linux and can be easily upgraded to run an alternative piece of firmware. I wanted to concentrate my network devices on the Linksys router and then run a single connection from the Linksys router to an Ethernet port on the back of the 2-Wire broadband router. I also wanted to make use of the fine bandwidth reporting available from the Tomato firmware which this router can be upgraded to run. The process of upgrading the router took about 10 minutes as it can be done from a menu option within the native Linksys firmware. Once Tomato was up and running on the Linksys it was easy to configure it to provide a good quality wireless network that replaced the old 2-Wire network and as an added benefit was also faster.

I configured the Linksys to store its bandwidth logs on a network Windows share and to forward its packet logs to WallWatcher as before and the results bandwidth-24hrswere immediately interesting. Tomato hosts a number of web pages that show bandwidth over varying periods of time. I could see straight away that the download bandwidth on the WAN port was considerably higher than I would have expected and at quite a sustained value (see the image on the right). The graph shows consistently high volume of download overnight and then a period of very low activity in the morning when all of the PCs were switched off followed by high usage again from about 2pm when they were restarted. Tomato also hosts a real time bandwidth display. Using these displays combined with WallWatcher it was easy to identify the PC responsible for the heavy usage and by examining the addresses of the remote end-points shown by WallWatcher it was also easy to determine the offending program.

TV Tonictvtonic_realtime

In my case, the program generating the traffic was the TV Tonic RSS service, a program responsible for downloading video podcasts from the Internet. I hadn’t realised that the program was still active as I had not made use of the client for a number of weeks. Incidentally, the TV Tonic client runs as an add-on to Windows Media Center (under Vista) and is quite a nice addition. Had I looked a little more closely at its configuration I would have noticed that not only did it have an option to limit the download bandwidth but also it had a download scheduler to control the time of day that it be allowed to download at all (both of these options would be nice to see adopted by the BBC iPlayer). I’m not quite sure why TV Tonic was downloading such large amounts of data but, not wishing to experience another month of bandwidth restriction, I immediately disabled the TV Tonic service and the Tomato real time monitor showed the corresponding reduction in bandwidth usage.

BBC iPlayerwallreviewer-out

Even after disabling the TV Tonic RSS service there still seemed to be a lot of network activity from my PC although I wasn’t running any obvious client program. A closer look at the WallWatcher log display showed a large number of incoming and outgoing UDP packets wallwatcher-iplayerbeing sent to external machines. WallWatcher comes with a charting tool called WallReviewer which gives a useful interactive graphical picture of incoming and outgoing traffic information over a given period of time. The WallReviewer chart of “Outbound Leaks by Remote Names” showed a large number of packets being sent to the machines iplaykdms82.telhc.bbc.co.uk and iplaykdms6.telhc.bbc.co.uk. The names of these remote sites suggested the BBC iPlayer might be responsible but the application wasn’t running and the option “allow programmes to be shared when you exit download manager” was not ticked in the iPlayer configuration dialogue so I had assumed that there ought to be no networking activity from the iPlayer Kontiki-based software. I found that if I disabled the Windows service named “KService” (which runs the BBC iPlayer program “C:\Program Files\Kontiki\KService.exe”) then all of this network activity stopped immediately. From the WallWatcher display it was clear to see that these packets were being sent about every 2 to 4 seconds but WallWatcher is not able to give any indication of the size of the packets.


To get a better indication of packet sizes a protocol analyzer is required. The “old faithful” in this area used to be called Ethereal but development on Ethereal has now been moved to Wireshark. Wireshark is simple to install and can be used on many platforms. It is also free and licensed under the GNU General Public License. There is a lot more to Wireshark that the casual user is ever likely to need and a basic knowledge of networking protocols and terminology helps but there is plenty of documentation.wireshark-iplayer

Running Wireshark on my PC confirmed that data packets were being sent to the BBC domain every two to four seconds but also showed that the packet sizes were small, 16 bytes of payload which by the time they have been wrapped in the UDP and IP packets amount to a 58 byte Ethernet frame. I find that having disabled the KService service I am unable to start up the BBC iPlayer but as soon as I re-enable the service the iPlayer functions as normal.


Having made these networking changes I am now in a much better position to know exactly how much traffic is being downloaded (or uploaded) over my broadband line and also able to detect this traffic early on to avoid triggering any ISP penalties. The tools required to monitor bandwidth are not expensive (in fact they are free) and are easily configured. I think that my ISP should have been able to give me the information that I needed to monitor and control my bandwidth – it feels a little like having been sold a car which has no fuel gauge.

One lesson that can be learnt from all of this is that it is becoming more and more important for anyone with a reasonable grasp of networking to take matters into their own hands to monitor their own network usage. I don’t see the ISPs relaxing their grip on our usage patterns in the short term, at least not until their own issues of congestion have been addressed. So by tightening up on wasted bandwidth we should be left with more to do the things that we really want to use it for.

April 13, 2008

Fair Use for BT Unlimited Broadband Traffic Shaping

Filed under: Media, Technology — Tags: , , , , — vextasy @ 8:27 am

gn_logoHave you ever noticed large variations in your broadband performance? If so, there are a number of factors you might want to check before putting it down to bad luck. It may be that you, like me, are having your download bandwidth silently restricted by your ISP.

My broadband is supplied by BT and known as BT Business Broadband Share, I’ve been a BT customer for a number of years now and I think the equivalent (and, I notice, somewhat cheaper) current package is known as BT Business Total Broadband – Option 3. Both packages are advertised with the term ‘unlimited usage’ and both refer you, in the small print, to BT’s fair use policy.

I wouldn’t describe myself as a heavy user of broadband. I think I may have purchased and downloaded a dozen music files from the Internet, in my life, and I have used the BBC iPlayer to download a similar number of TV programmes, although I was careful to tick the little box that prevented the iPlayer from re-sharing those files after I had finished watching them. I download, perhaps, 6 large DVD size images from Microsoft’s MSDN subscriber site a year as part of my developer network subscription. I connect to the office with a VPN connection to collect email and occasionally use remote desktop or VNC to connect to one of several remote servers to perform evening or weekend maintenance. Other than that, I browse the web and listen again to a repeat of a Radio 4 audio programme on average about once a month. None of these things are particularly expensive in terms of bandwidth.

We are only a short distance from our local exchange and so usually comfortably achieve download bandwidth figures of over 6M bits/sec, but recently I had noticed much longer delays in displaying web pages from all sources and interrupted video streaming. We have a number of other PCs in the house on our wireless LAN and they were all experiencing similar problems so I checked the router (a BT supplied 2-Wire 1800) and noted that its broadband monitor showed low download and upload demand. This made me suspicious that there was a problem with the wireless network itself and so I checked the various settings, rebooted PCs and restarted the router and all of this made no difference at all to the download performance. I even chose to accept the firmware upgrade that the router was offering me in the hope that it might fix the problem but, rather ironically, the only visible difference I could detect was that the router’s nice bandwidth monitoring page has now been removed which means that I no longer have any indication of the upload or download bandwidth in use at any given point in time.

Noting that the download bandwidth was low, and knowing that our line normally performs well, I assumed then that the problem might be congestion at the exchange. There are good congestion checking tools at nildram and plusnet and plenty of information about broadband exchanges at samknows.com. These resources all suggested that my local exchange had no congestion issue.

Puzzled, I thought I’d monitor the bandwidth and see if I could determine a pattern. I initially suspected some form of interference from, say, a poor electrical connection or a fluorescent light, both of which can have this effect on broadband speeds, or so I had read.

You can check you broadband bandwidth with the excellent speed checker at speedtest.net which allows you to maintain a nice record of the measurements you have taken over a period of time or the less impressive BT offering at speedtester.bt.com which I found had to be run with administrative privileges on my Vista system but which does additionally provide you with what BT call your IP profile. There is an excellent description of this IP Profile at kitz.co.uk and a wealth of background information about ADSL too.

According to speedtester.bt.com my IP profile was 6.5Mbits/sec which was what I had expected:

IP profile for your line is - 6500 kbps
DSL connection rate: 448 kbps(UP-STREAM)  7616 kbps(DOWN-STREAM)
Actual IP throughput achieved during the test was - 2569 kbps

But depending on the time of day that I ran the test the IP throughput would vary from the expected 6340 kbps to as low as 379 kbps in the evening, rising through 3788 kbps after 11pm back to full speed after midnight and during the daytime. I saw this pattern repeat itself over a number of days.

Armed with this information I began to become suspicious that my line was being traffic shaped in some way. I called BT Business Broadband support and the nice lady there confirmed that my number was being restricted but she seemed surprised that I had not received an email alerting me to this. Broadband support said they were unable to give me any more information other than to say that it was due to excessive usage and gave me a phone number to ring to investigate further. That number turned out to be for reporting security breaches but they, in turn, gave me the email address: liteusage@btbroadbandoffice.com to which issues regarding traffic shaping and bandwidth restriction can be addressed.

I understood from the phone conversation that it was BT’s policy to review these restrictions at the billing points and that the restrictions could be gradually reduced if not completely removed when the excessive usage ceased. This meant that I should expect restrictions to be in place for several more weeks until BT saw fit to restore my service.

I emailed a request asking if they could help me to:

  1. Understand why the restriction has been applied .
  2. Help me get it released as soon as possible as it is interfering
    with my business use of this line.
  3. Give me an idea of the volume (or nature) of downloads that must have
    been present to have this restriction applied so that I can ensure
    it doesn’t happen again.

and the reply I got back said:

From the description you have given it appears that you are being
traffic shaped under the terms of our Fair Usage Policy, this is
why your connection is slow between 5pm and midnight.

and then followed this up with the surprising statement:

I am sorry but we cannot lift this restriction for you as our suppliers implement these measures.

Neither of these was particularly helpful or acceptable so I thought I’d better have a closer look at the fair use policy to see what I had done wrong and that is where the confusion really begins. The fair use policy doesn’t explain what BT consider fair use to be. I’d be only to happy to make a considered judgement about whether my £50/month unlimited broadband package was worth that amount if I could see what I was allowed to use it for. The policy explains in its three major sections:

Why do we have a fair use policy?

BT explain that their “… Fair Use Policy manages inappropriate use and makes sure the service can be used fairly by everyone”. And they define inappropriate use:

“A very small number of our customers use their broadband service inappropriately, for example when sending or downloading very large files, or using ‘peer to peer’ and file sharing software (which may be sending and receiving video and other large files constantly)”

So their concern appears to be related to the transmission of very large files and “peer to peer” or file sharing software. There is no mention of an acceptable download (or upload) usage figure, especially for their unlimited services.

How does the fair use Policy work?

Quite simply:

“If you regularly use the service inappropriately during peak hours, and we believe this is unfairly affecting other customers’ use of the service, we’ll manage your bandwidth during peak times (which could result in reduced service speeds).”

The policy clearly states that BT will control bandwidth for what they deem inappropriate use (earlier rather feebly defined as something involving large files and, possibly “peer to peer” and file sharing software) if this is done regularly (again undefined). There is no indication of how long they will continue to manage bandwidth for or an advanced warning that it might be about to happen. They do say that:

“If you continue to use your service inappropriately we reserve the right to end your agreement with us and will give you notice before doing so.”

But, of course, if you weren’t aware that you were using the service in an inappropriate way to begin with how are you supposed to know that you have continued to use it inappropriately. This seems to me to be an opportunity for BT to silently manage bandwidth to whatever extent suits them whilst continuing to charge the full rate for the service.

How do I know if the Fair Use Policy affects me?

BT say: “Our Fair Use Policy applies to all our customers but it’ll only actually affect you if you’re one of the very few customers who make inappropriate use of our service”. So another recursive and incomplete piece of information.

But don’t worry because:

“If you don’t use peer to peer, file sharing or other inappropriate software and you’re not, for example, constantly downloading or uploading: videos or very large files, you’re unlikely to be affected by our Fair Use Policy.”

So quite simply, don’t trouble your pretty little heads over our fair usage policy as it is unlikely to affect you. But hang on a minute,

  • I use the BBC iPlayer and that is peer-to-peer file sharing software,
  • I pay several hundred pounds a year to Microsoft to allow me to download operating system DVD images or sometimes videos from their web site and those are large files.

So, on both those counts BT can legitimately claim that I have fallen foul of their fair use policy and without warning restrict my broadband connection to whatever extent they wish and for however long they wish and still continue to bill me the same amount of money even though they neither specify the acceptable usage limits or provide me with any mechanism by which I can determine my current usage?

That doesn’t seem like fair use to me.

April 8, 2008

ORDB – Gone But Not Forgotten

Filed under: Media, Technology — Tags: — vextasy @ 9:34 am

ORDB has posed another Web traceability problem for me. They recently announced that they were closing down their Open Relay database that has been used by mail servers to help identify incoming mail likely to be considered spam based on knowledge of the mail senders IP address; the idea being that if enough people reported spam to a centralised organisation then a database could be maintained that could track IP addresses from which spam appeared to originate and then that organisation could provide a lookup service to mail servers which would allow them to rapidly check all incoming mail to determine if it originated from one of these blacklisted IP addresses.

The problem is that although I recall reading about their impending closure, there is now nowhere authoritative that I can find that confirms the current status; their website, at http://ordb.org/news/?id=38, where the original news item was posted (I believe) is currently offline and the word on numerous online sites is that the ORDB service is now reporting all IP addresses as being blacklisted in an attempt to force mail server administrators to remove the ORDB database from their list of blacklist services, presumably to reduce the load on their servers if they decide to re-open in some form. Looking through online forums and blog posts it is clear that plenty of people are experiencing the results of this change.

There is really no technical issue here for me – the correct solution is simply to remove ORDB from my mail servers, which is something I did a while ago anyway. The real issue is that when I googled for “ordb” it was apparent that most of the articles were based on the same sources at www.nabble.com and in turn Slashdot which reported the nabble.com message. The one site I did find that reported a conversation with a former ORDB operator was on a German site at www.heise.de which reported:

Andreas Plesner Jacobsen, a former ORDB operator, explained to heise online sister publication iX that this measure has been introduced because the zone is still swamped with queries. The intention is to get mail server operators to stop using ORDB. Simply deleting the domain was not a viable alternative, since the load would then merely be directed to the .org name server.

Ironically, it seems that the only way ORDB could get people to act to remove them from their mail servers was by breaking their service in a way which forced administrators to investigate – simply making an announcement on their web site was clearly not sufficient.

April 7, 2008

My 10 Favourite (free) Windows Tools of all Time.

Filed under: Software — Tags: , , , , , , , , — vextasy @ 9:57 pm

Reading Ed Bott’s postings about his and his readers’ favourite Windows programs of all time I was surprised to note just how many of the programs on the list had an associated price tag rather than being free (as in beer). In particular, what attracted my attention was that had I been asked to guess which were free and which were not I would probably have failed miserably. For example, a text editor for $33, a note taking tool for $60 and a screen capture utility for $40, but a complete news aggregator for free.

I work, mostly, in a Microsoft environment and so the majority of my main software development tools for that platform are either purchased or licensed through an (expensive) subscription but, like most readers, I like to adorn that environment with utilities that make for a more agreeable working experience. Sometimes those utilities relate directly to work tasks and sometimes less so, but what I notice is that most often those utilities are free (or effectively so – more on this later).

I constructed a list of the utilities that I use on a regular basis at work and at home and very quickly the list grew well beyond 10 in size. As it doesn’t seem sensible to attempt to order them in any way (because such an ordering would make an assumption about your motives for having them in the first place) I leave them unordered. Likewise, as I don’t feel comfortable choosing my top ten, I describe more than that number here but the real list is much longer and growing.


TrueCrypt is disk encryption software which allows either an entire disk partition to be encrypted or else a virtual encrypted disk to be created from a file and then mounted as a Windows drive. The software is Open Source, well documented and thoroughly well thought out. I haven’t had the courage to get to encrypt a real partition yet but do use it to maintain a number of well protected virtual drives that I can mount when I need access to the documents that I store securely inside them. A drive can be mounted once the required password (or password and key file, or correct encryption keys) are provided and once mounted it can be used just like any other Windows drive. The contents of a TrueCrypt drive are never stored in their decrypted state on disk they are only ever held temporarily in RAM. TrueCrypt drives are a great place to store that collection of documents that you know should really be kept secure.


For software developers, like me, who were brought up in a Unix environment the lack of a real command line in the Windows environment can be stifling. Now I know (the awfully named PowerShell) is now available, but what made the Unix environment so complete was the rich set of commands that could be glued together with whichever variant of the Bourne shell was in vogue. Cygwin provides that same environment but hosted under Windows. The choice of programs is truly massive: editors, shells, compilers, interpreters, text and document processors, libraries, windowing systems. Most things GNUish can be found there courtesy of the GNU C compiler and friends too. Integration with Windows through the filesystem means that all of these tools can be used to process files and media residing on any Windows drive.

TimeSnapper TimeSnapper1

TimeSnapper quietly records your activity by taking snapshots of your computer screen at regular intervals through the day. The interval between snapshots can be configured to a given number of seconds and the recording is achieved without any noticeable pause or flicker. This is really handy on those days where you have moved from one task to another and have not been as meticulous about recording your exact timings as you should have been as it allows you to replay the day a snapshot at a time or to jump quickly to a particular time of day and see what you were working on at that point in time. You provide TimeSnapper with a folder it can use to store the snapshot images and chose the format (.png, .jpg, .gif, .wmf, .tiff, .bmp, .emf) and the resolution of the stored images as a percentage of the full screen resolution. TimeSnapper will also manage the archiving of the snapshots if you provide it with an age beyond which you wish it to delete old images or an upper limit to the amount of space you would like it to allocate to storage. Multiple displayed are handled too. This is a tool you can forget about until you need it, and then its a lifesaver.

Copernic Desktop Searchcds2-screenshot-all-big

Copernic Desktop Search is one the many similar search products but what I really like about this program is its intuitive interface. Of course, it is packed with all of the features you would expect from any such search tool and, of course, it indexes a myriad of document and media file formats inspecting meta data inside the files for rapid lookup. It also understands, and so can index, email and contact information from Outlook, Outlook Express, Eudora and Mozilla Thunderbird.

The using interface, rather than relying on a web browser as some search tools do, reacts dynamically as you type, homing-in on the information being sought. Indexing happens on-the-fly and only when the machine is not heavily loaded (and this is configurable). Copernic confirm on their web site that you can “Rest assured that the data indexed by CDS stays on your PC and on no account will it be transferred to us or any of our partners”. The licence only allows for non-commercial use. A separate licence exists for commercial application. That said, I know people for whom this has revolutionised the way they use their PC and I recommend this as a productivity tool.

Virtual CloneDrive virtualclonedrive

As a software developer I find that I am often presented with application software in ISO format. It is always a pain to have to burn a DVD just so that it can be mounted in a Windows drive and then discarded and probably never used again (probably never even labelled) once the installation has been completed. Most of my MSDN software arrives this way. SlySoft’s Virtual CloneDrive allows these images to be mounted directly from the ISO file on the file system. Several other formats are supported in addition to ISO.

MusicBrainz Picardpicard

If you have ripped your CD collection to MP3 or other digital format you will almost certainly have found errors in the track and album metadata that the music files contain or inconsistencies in the naming conventions used by each of the different people who have provided this information. MusicBrainz Picard comes to the rescue by applying the accumulated knowledge from the very well moderated MusicBrainz database. MusicBrainz is a community music metadatabase that attempts to create a comprehensive music information site and you can use the Picard tagger to automatically identify digital music and then tag it and to clean up the existing metadata tags in your digital music collection. I used Picard to correct the Windows Media Player created tags in my own music library when I ripped my entire CD collection to mp3 format and use it regularly each time I purchase music.

Pidgin logo.pidgin

Pidgin is a multi-protocol messaging client that handles a large number of instant messaging protocols: AIM, Bonjour, Gadu-Gadu, Google Talk, Groupwise, ICQ, IRC, MSN, MySpaceIM, QQ, SILC, SIMPLE, Sametime, XMPP, Yahoo!, Zephy. I can really only claim to have used the MSN and IRC protocols but the reason for turning to Pidgin was to allow me to communicate with my family members on MSN without having to endure advertisement hell. Pidgin supports away messages, typing indications and file transfer between clients.

Firebug firebug

If you are anything more than the most casual of Firefox users or if you create any kind of HTML content or even if you are simply interested in the structure of the HTML page that you are viewing in Firefox you should be interested in the Firebug extension to Firefox. Firebug integrates with Firefox to enable rich examination of a web page structure including:

  1. an interactive and graphical identification of the effect of individual sections of HTML on the resulting display going from both HTML to display and from display to HTML.
  2. an indication of the CSS rules, and the order in which they have been applied, that determine the final appearance of a screen element.
  3. the ability to change elements of the CSS or HTML source and immediately see the resulting effect on the display.

Firebug was written by one of the original Firefox developers and the slickness of the integration is evident. If I could only keep one Firefox extension it would be Firebug.


Its difficult to get to excited about a file archive tool, especially one that performs well is unobtrusive and just gets the job done. 7Zip is just that kind of tool, integrating well with the Windows explorer shell context menus but providing more functionality and better performance than the native Windows archiver (Compressed folders). When writing an archive, 7z, ZIP, GZIP, BZIP2 and TAR target formats are available and when reading an archive any of RAR, CAB, ISO, ARJ, LZH, CHM, MSI, WIM, Z, CPIO, RPM, DEB and NSIS formats are available. 7Zip can optionally apply AES-256 encryption when creating 7z and ZIP format archives.


Gimp, the GNU Image Manipulation Program, is as close as you will get to a tool like Photoshop or Paint Shop Pro without spending a lot of money. For most of the image related tasks that I need to perform it is overkill (by a long shot too) but if you are prepared to put some time into learning the basic techniques some impressive results can be obtained. There are quite a lot of helpful web sites within reach of Google that contain hints, tips and tutorials for those who make the effort. Also take a look at paint.net, a relative newcomer but receiving a lot of praise.


JungleDisk is a tool that puts a user-friendly front-end on top of Amazon’s S3 Storage Service. S3 enables inexpensive off-site storage of files up to 5GB in size to an unlimited capacity. Storage costs are of the order of $0.18 / month per GB with data transfer rates of between $0.10 and $0.20 per GB. JungleDisk itself is not free (in spite of my claim in the title of this post), it costs $20, but can be used on as many PCs as you like with the same Amazon S3 account. I include it here because compared to the cost and worry of on-site storage the combined cost of JungleDisk and Amazon S3 is effectively free, at least as far as I am concerned. JungleDisk can perform on-the-fly encryption of data as it travels from the PC to S3 and decryption on its return journey, it can make the S3 storage appear as a mapped local drive and it can perform scheduled backups from the PC to S3.

Launchy launchy_icon

Launchy is a smart search program which tries to guess which program you are looking for and will launch it with the minimum number of keypresses required to satisfactorily identify the desired program. It is designed to help you forget about your start menu, the icons on your desktop, and even your file manager.

This is a utility that I didn’t expect to survive my move from Windows XP to Windows Vista because at first glance it appears to provide much the same functionality that is now found with the search facility that is built into Vista’s Start Menu search box. Indeed, to begin with, I survived without it for a couple of months but then I began to miss the fact that Launchy is started with only a hot-key combination and requires no mouse movement or clicks. Launchy lurks in the background and responds to the Alt-spacebar key sequence by opening a small input field to accept keyboard input. On typing, Launchy searches its indexed list of known programs for the closest match, the search being refined with each additional keypress. When the desired program is identified a hit of the enter key is all that is required to launch the program.

Launchy can be customised to search specific locations for commands and to recognise additional files type, or to provide additional arguments or accept user supplied parameters to commands and it can also perform online searches with google, msn, yahoo, live, weather, amazon, wikipedia, dictionary, thesaurus, imdb, netflix, and msdn.


SharpReader is an RSS feed aggregator created by Luke Hutteman and is the only RSS reader that I have ever been completely comfortable with. The application is infrequently updated but (possibly as a result) runs without faulting and simply does the job well. In addition to allowing a collection of feeds to be browsed it also presents a stacked list of alerts up the right hand edge of the display whenever new items arrive. The lifetime of these alerts can be adjusted to allow just enough time to quickly scan them without them becoming too much of a distraction to the job in hand. At the time that I started using SharpReader the only other utility that I felt came anywhere close to it was FeedDemon. FeedDemon is now also a free product and I have been dual running it alongside SharpReader – the jury is out, but SharpReader still has the edge.


Vi was one of the first Unix visual text editors, taking its name from the two character command that switched its predecessor ex into a, so called, visual mode. Ex, in turn, is a descendent of ed which was written by Ken Thomson back in the 1960s as part of the Multics environment and contained one of the first implementations of regular expressions. Vim was created in 1991 for the Amiga computer as an extended version of the vi editor and gVim is the graphical variant of vim. The expressiveness of regular expressions combined with the rather terse but necessary and sufficient approach command driven editing that this family of editors supported went on to fuel many of the ideas in other important Unix commands, notably grep, sed and later awk (which you could argue was responsible for the creation of Perl). The lineage continues with Rob Pike’s sam and acme for the Plan 9 and Inferno operating systems.

You could argue that Vim is part of Cygwin which I have described elsewhere but I think it deserves to be singled out here if for no other reason than for the fact that on a Windows system it allows you to replace the hopeless notepad with something that at least allows you to perform some useful tasks and, if you are prepared to make the effort to learn its command syntax, become more productive too.

March 30, 2008


Filed under: Media, Software — Tags: , — vextasy @ 10:45 pm

MusicBrainz is a community-driven music meta-database. It is almost a Wikipedia of the music world. Content is maintained by a community in which changes have to be approved by a vote before they become permanently accepted into the database and a team of dedicated moderators oversee the whole thing. The words orderly and consistent spring to mind; emphasis is placed on consistency of style (by that I mean, for example, capitalisation and abbreviation conventions) and correctness. In return for their input, community members are rewarded with a great tool for maintaining their own music databases – the tags in their own music collection. The tool is called Picard and is free to download. In acknowledgement of the quality of the database MusicBrainz has now been licenced by an impressive list of customers including MusicIP, The British Broadcasting Corporation (BBC) and Last.fm.
MusicBrainz Logo
At some point during the Christmas holiday of 2005 I began the process of ripping my CD collection to allow it to be recalled and replayed in a more controlled manner and to reduce the amount of space that it occupied. I opted to make it visible on my home LAN using a combination of a Linksys NSLU2 (affectionately known as a SLUG) and a 250GB Buffalo DriveStation. The DriveStation is a USB hard drive and the Linksys bridges such a USB drive to an ethernet network. The Linksys will support two drives, although I’ve not needed to use both here.
The nice thing about the Linksys is that it runs Linux and can be customised to perform a number of task in addition to file serving. Both devices sit quietly tucked out of the way in an upstairs room and are directly connected to my wireless broadband router. The Linksys runs the ubiquitous (at least in the Unix world) Samba SMB file server which allows it to look just like a networked PC to other PCs on the home network. Files can be accessed from its drive(s), given the appropriate permission, as if they were on a PC but, of course, there is no fan noise or hot processor or display and so power consumption is at a minimum. The beauty of this arrangement is that they can be left running and so are always available.

I used Windows Media Player to rip the, roughly, 5,700 tracks from CD to mp3 format. Media Player makes a great attempt to tag the mp3 files correctly but for an irritatingly large number of tracks the information is either incorrect or inconsistent. This is where MusicBrainz comes to the rescue. Specifically, MusicBrainz Picard, their free and open source, cross-platform music file tagger.

Picard uses the MusicBrainz database to correctly and consistently tag mp3, wav, vorbis, flac, mpc, mp4 and wma format files. If asked to identify a music CD it will recognise the artist and release based on an analysis of the content of the CD which it uses to construct a unique disc-id which can be compared to known disc-ids in the MusicBrains database (at the time of writing there are approximately 228,000 such known disc-ids). Alternatively, Picard can recognise individual music files by a form of audio fingerprinting and makes a special effort to associate clusters of music files with a particular release or album. If neither of these techniques succeed the GUI allows manual associations to be made with the correct titles from the database.

Once associations have been made, Picard displays the tag information currently stored in the music file alongside the suggested information (from the MusicBrainz database) together with an indication of closeness of fit and allows (selective) correcting of the tags in the music file. Plugins to Picard allow you to pull down cover art of incorporate genre information from Last.fm but I haven’t tried either of these.

Reading Ian Dixon and Ed Bott’s postings on how they organise their music collections made me realise just how many different ways there are to achieve the same outcome. Where I think the MusicBrainz tools score is in the quality of the database that sits behind them.

March 29, 2008

BBC iPlayer Installation Woes and Success

Filed under: Media — Tags: — vextasy @ 12:02 am

I’ve seen several postings about issues people have had with BBC iPlayer installations and many more posts containing descriptions of attempts to solve these issues. In general, I found that installation on three machines (two Vista and one XP) in the house was very straightforward but did have a problem on one of the three machines, a Vista machine.

I believe my problem may have been caused by running the installation program as administrator but, whatever the cause, the symptoms were that I could download programs with the download manager but when I attempted to play any program by clicking on “Watch Now” the iPlayer would simply silently refuse to do anything. If I modified the iPlayer settings to allow me to use “my default media player” (in my case Windows Media Player) it would also fail to play the video and (if I recall correctly) display a message about being unable to open or find the media.

The solution in my case was entirely related to file and folder permissions and the following steps resolved everything:

  1. Find the folder where the iPlayer stores the downloaded media files. In my case, C:\Users\Public\Videos\My Deliveries\iplayer_live
    This information may be determined from the iPlayer settings page.
  2. Look at the permissions on one of the media files in the folder. The files will have names like 8780437_74568151_Selection_DOWNLOAD.wmv.
    Right click on a file and choose properties->security and check that your username (or everyone) has at least “read & execute” and “read” permission on the file.
    If it doesn’t then it is quite likely that this will be your problem and so continue with the steps below.
  3. On a Vista machine for which the installation worked first time I find that the files have “read & execute” and “read” permission
    for my username and “full control”, “modify”, “read & execute”, “read”, and “write” permissions for my group (Administrators).
    You will need to manually add sufficient permissions to allow you to at least read the file. I think I took the heavy handed approach that
    we all take in times of desperation and applied “full control” to the .wmv and its accompanying .smi file (the file with the same name
    but different file suffix).
  4. Test the file, by either right clicking the .wmv file and choosing “play”, or by clicking the corresponding “Watch Now” link
    in the iPlayer “My Downloads” window.
  5. If the file doesn’t play (allow a suitable startup time) then this isn’t the solution to your problem.
  6. If the file does play, then you will most likely need to perform a similar fix for all of your downloaded programs.

In my case, the real problem lay in the permissions associated with the folder C:\Users\Public\Videos\My Deliveries\iplayer_live.
My solution was to change the permissions on the folder to allow “full control” to “Everyone” and to ensure that such permissions were subsequently inherited by all new files that the iPlayer download manager created in the future.

December 12, 2006

Hello (WordPress) World!

Filed under: Uncategorized — vextasy @ 1:02 pm

This post marks the point that parts of the Radio Userland blog were moved to WordPress.

February 26, 2006

Who Can You Trust?

Filed under: Media — vextasy @ 10:46 pm

The following two snippets of information were taken from Tom Flynn’s Did You Know? column, part of the Point of Enquiry podcast from the Center for Inquiry:

According to a June 2005 Gallup pole: 73% of Americans believe in the paranormal, 41% believe in ESP, 37% believe that houses can be haunted by ghosts, 25% believe in astrology and 42% believe that people can sometimes be possesed by the devil. According to a later Gallup pole: 24% of Americans believe that Alien beings have visited Earth. According to the Roper Center for Public Opinion at the University of Connecticut: 4 million Americans have reported that they have been abducted by aliens.

The second largest belief group in America is the non-religious. The total number of individuals who are not associated with a religious group in America is roughly 48 million, about 16% of the total population. They outnumber every other faith tradition except Christianity. This is more than the number of African Americans or Gays and Lesbians in America. Most of these unaffiliated (roughly 30 million people, which is about 10% of all Americans) are secular, humanist, atheist or agnostic but there is only one openly agnostic, aetheist or secular congressman or senator.

Both contain some interesting facts. But how do we know they are true? Tom names his sources, in this case Gallup and the Roper Center but that’s as far as a listener or a web page reader can really get towards verifying the sources without paying to see the actual surveys.

But this is nothing new. Whenever anyone tells us a fact we must make some judgement about the truthfulness of their statement and that is exactly what we do when reading a website or listening to a podcast. In my experience, the following weighs heavily when making such

  1. Who else endorses the podcast or web site?
  2. Over a period of time what proportion of their productions remain within my own determined bounds of credibility?

On the first count, both Richard Dawkins and Martin Gardner have contributed and so have, in some sense, endorsed the organisation. Both men are respected academics in their respective fields of biology and mathematics and that is enough to satisfy me for the moment.

On the second count, only time will tell.

February 4, 2006

Dear Valued Customer

Filed under: Technology — vextasy @ 12:44 am

I have been a very loyal customer of DynDns for quite sometime now, but a few days ago the following email dropped into my mailbox:

Dear Valued Customer:
The hostname, slug.xxx.co.uk, in account ABCDEF, has
been blocked for abuse. This action has been taken due to the
receipt of multiple updates originating from the same IP

The message was from DynDns and was sent in response to the actions of my incorrectly configured DNS update client. What surprised me was just how taken aback I was at the use of the term abuse. As someone who has been using the Internet since some time before the introduction of the web browser I was shocked that I could be accused of abuse. Surely there was some mistake. Nevertheless they were absolutely right.

According to dictionary.com:

1. To use wrongly or improperly; misuse: abuse alcohol; abuse a privilege.
2. To hurt or injure by maltreatment; ill-use.
3. To force sexual activity on; rape or molest.
4. To assail with contemptuous, coarse, or insulting words; revile.
5. Obsolete. To deceive or trick.

And I was certainly using their service wrongly. The problem was in the way that I had configured my SLUG to update its own entry on DynDns so that I could always find it from work by its fully qualified domain name even if my non-fixed home IP address changed. My mistake was in not making its update rules quite strict enough. I was occasionally updating my DNS entry when it hadn’t really changed and so I was violating the DnsDns Update Abuse Policy.

The best things may be abused.
– The Penguin Dictionary of Proverbs.

I’ve fixed the problem now and hopefully I’m no longer an abuser but a valued customer once again.

January 29, 2006

BBC Podcasts – December 2005

Filed under: Media — vextasy @ 10:50 pm

Earlier, I made reference to the published BBC podcast figures for July 2005. The BBC do not distinguish between on-demand streamed programmes and podcast downloads in their ongoing download and podcast trial. The figures for December 2005 are now available on the online statistics page and show that the monthly downloads figure for the BBC Radio 4 In Our Time programme has grown from 25,000 to 145,000 in the six months leading up to December 2005.

That’s quite a jump.

« Newer PostsOlder Posts »

Create a free website or blog at WordPress.com.