Twitter Ye Not (there’s #uksnow time to lose)

I feel like I’ve just seen a meteor shower – an unexpected but surprisingly satisfying event took place last night. The micro-blogging site Twitter receives mixed reviews. Many people love it, are perhaps even a little addicted to it, others fail to see what all the fuss is about. Whatever your views there is no doubt that it has spawned a number of interesting discussions about the future directions of blogging and social networking.

Several tools have formed around twitter which aim to apply some form of structuring to the naturally unstructured twitter message (a tweet). One such tool is #hashtags which filters all tweets that contain a hashtags pattern (a word that begins with a hash character) and presents them as a chronological list of tweets collected under the heading of the given hashtag thus enabling communities of twitterers to follow a stream of conversations about a common topic. There is no organization to all of this, but rather a community that feeds off a number of sites that make use of this data such as twitterfall.com, search.twitter.com, twist.flaptor.com and hashtags.org itself.

Last night a tag formed to discuss the amount and location of snow that was falling in the UK. Although the conversation was informative it was often difficult to make use of the information because tweeters were inconsistent in declaring their location and the amount of snow they were experiencing. At some point in the evening one twitter user decided to build a mashup that took the #uksnow stream of tweets and parsed their content for the sequence

#uksnow postcode score/10

Where postcode is replaced by the first portion of the tweeters UK postcode and score is a rating (out of ten) used to indicate the amount of snow that is currently falling in that postcode region. The mashup combined this information with google maps, plotting in white regions with the heaviest amounts of snow. It was observed by one twitter user just how consistent this map was with the MET Office radar but with the advantage of being updated in real time.

Clearly there are obvious weaknesses, such as the subjective nature of a  zero to ten snow scale (although the, once popular,  Beaufort scale for measuring wind speed springs to mind) and the influence of the natural differences in population density which would bias the influence of the tweets in favour of heavily (twitter) populated areas of the UK, but I can’t help feeling that I witnessed something that was both spontaneous and surprisingly useful.

Fair Use for BT Unlimited Broadband Traffic Shaping

Have you ever noticed large variations in your broadband performance? If so, there are a number of factors you might want to check before putting it down to bad luck. It may be that you, like me, are having your download bandwidth silently restricted by your ISP.

My broadband is supplied by BT and known as BT Business Broadband Share, I’ve been a BT customer for a number of years now and I think the equivalent (and, I notice, somewhat cheaper) current package is known as BT Business Total Broadband – Option 3. Both packages are advertised with the term ‘unlimited usage’ and both refer you, in the small print, to BT’s fair use policy.

I wouldn’t describe myself as a heavy user of broadband. I think I may have purchased and downloaded a dozen music files from the Internet, in my life, and I have used the BBC iPlayer to download a similar number of TV programmes, although I was careful to tick the little box that prevented the iPlayer from re-sharing those files after I had finished watching them. I download, perhaps, 6 large DVD size images from Microsoft’s MSDN subscriber site a year as part of my developer network subscription. I connect to the office with a VPN connection to collect email and occasionally use remote desktop or VNC to connect to one of several remote servers to perform evening or weekend maintenance. Other than that, I browse the web and listen again to a repeat of a Radio 4 audio programme on average about once a month. None of these things are particularly expensive in terms of bandwidth.

We are only a short distance from our local exchange and so usually comfortably achieve download bandwidth figures of over 6M bits/sec, but recently I had noticed much longer delays in displaying web pages from all sources and interrupted video streaming. We have a number of other PCs in the house on our wireless LAN and they were all experiencing similar problems so I checked the router (a BT supplied 2-Wire 1800) and noted that its broadband monitor showed low download and upload demand. This made me suspicious that there was a problem with the wireless network itself and so I checked the various settings, rebooted PCs and restarted the router and all of this made no difference at all to the download performance. I even chose to accept the firmware upgrade that the router was offering me in the hope that it might fix the problem but, rather ironically, the only visible difference I could detect was that the router’s nice bandwidth monitoring page has now been removed which means that I no longer have any indication of the upload or download bandwidth in use at any given point in time.

Noting that the download bandwidth was low, and knowing that our line normally performs well, I assumed then that the problem might be congestion at the exchange. There are good congestion checking tools at nildram and plusnet and plenty of information about broadband exchanges at samknows.com. These resources all suggested that my local exchange had no congestion issue.

Puzzled, I thought I’d monitor the bandwidth and see if I could determine a pattern. I initially suspected some form of interference from, say, a poor electrical connection or a fluorescent light, both of which can have this effect on broadband speeds, or so I had read.

You can check you broadband bandwidth with the excellent speed checker at speedtest.net which allows you to maintain a nice record of the measurements you have taken over a period of time or the less impressive BT offering at speedtester.bt.com which I found had to be run with administrative privileges on my Vista system but which does additionally provide you with what BT call your IP profile. There is an excellent description of this IP Profile at kitz.co.uk and a wealth of background information about ADSL too.

According to speedtester.bt.com my IP profile was 6.5Mbits/sec which was what I had expected:

IP profile for your line is - 6500 kbps
DSL connection rate: 448 kbps(UP-STREAM)  7616 kbps(DOWN-STREAM)
Actual IP throughput achieved during the test was - 2569 kbps

But depending on the time of day that I ran the test the IP throughput would vary from the expected 6340 kbps to as low as 379 kbps in the evening, rising through 3788 kbps after 11pm back to full speed after midnight and during the daytime. I saw this pattern repeat itself over a number of days.

Armed with this information I began to become suspicious that my line was being traffic shaped in some way. I called BT Business Broadband support and the nice lady there confirmed that my number was being restricted but she seemed surprised that I had not received an email alerting me to this. Broadband support said they were unable to give me any more information other than to say that it was due to excessive usage and gave me a phone number to ring to investigate further. That number turned out to be for reporting security breaches but they, in turn, gave me the email address: liteusage@btbroadbandoffice.com to which issues regarding traffic shaping and bandwidth restriction can be addressed.

I understood from the phone conversation that it was BT’s policy to review these restrictions at the billing points and that the restrictions could be gradually reduced if not completely removed when the excessive usage ceased. This meant that I should expect restrictions to be in place for several more weeks until BT saw fit to restore my service.

I emailed a request asking if they could help me to:

1. Understand why the restriction has been applied .
2. Help me get it released as soon as possible as it is interfering
   with my business use of this line.
3. Give me an idea of the volume (or nature) of downloads that must have
   been present to have this restriction applied so that I can ensure
   it doesn’t happen again.

and the reply I got back said:

From the description you have given it appears that you are being
traffic shaped under the terms of our Fair Usage Policy, this is
why your connection is slow between 5pm and midnight.

and then followed this up with the surprising statement:

I am sorry but we cannot lift this restriction for you as our suppliers implement these measures.

Neither of these was particularly helpful or acceptable so I thought I’d better have a closer look at the fair use policy to see what I had done wrong and that is where the confusion really begins. The fair use policy doesn’t explain what BT consider fair use to be. I’d be only to happy to make a considered judgement about whether my £50/month unlimited broadband package was worth that amount if I could see what I was allowed to use it for. The policy explains in its three major sections:

Why do we have a fair use policy?

BT explain that their “… Fair Use Policy manages inappropriate use and makes sure the service can be used fairly by everyone”. And they define inappropriate use:

“A very small number of our customers use their broadband service inappropriately, for example when sending or downloading very large files, or using ‘peer to peer’ and file sharing software (which may be sending and receiving video and other large files constantly)”

So their concern appears to be related to the transmission of very large files and “peer to peer” or file sharing software. There is no mention of an acceptable download (or upload) usage figure, especially for their unlimited services.

How does the fair use Policy work?

Quite simply:

“If you regularly use the service inappropriately during peak hours, and we believe this is unfairly affecting other customers’ use of the service, we’ll manage your bandwidth during peak times (which could result in reduced service speeds).”

The policy clearly states that BT will control bandwidth for what they deem inappropriate use (earlier rather feebly defined as something involving large files and, possibly “peer to peer” and file sharing software) if this is done regularly (again undefined). There is no indication of how long they will continue to manage bandwidth for or an advanced warning that it might be about to happen. They do say that:

“If you continue to use your service inappropriately we reserve the right to end your agreement with us and will give you notice before doing so.”

But, of course, if you weren’t aware that you were using the service in an inappropriate way to begin with how are you supposed to know that you have continued to use it inappropriately. This seems to me to be an opportunity for BT to silently manage bandwidth to whatever extent suits them whilst continuing to charge the full rate for the service.

How do I know if the Fair Use Policy affects me?

BT say: “Our Fair Use Policy applies to all our customers but it’ll only actually affect you if you’re one of the very few customers who make inappropriate use of our service”. So another recursive and incomplete piece of information.

But don’t worry because:

“If you don’t use peer to peer, file sharing or other inappropriate software and you’re not, for example, constantly downloading or uploading: videos or very large files, you’re unlikely to be affected by our Fair Use Policy.”

So quite simply, don’t trouble your pretty little heads over our fair usage policy as it is unlikely to affect you. But hang on a minute,

• I use the BBC iPlayer and that is peer-to-peer file sharing software,
• I pay several hundred pounds a year to Microsoft to allow me to download operating system DVD images or sometimes videos from their web site and those are large files.

So, on both those counts BT can legitimately claim that I have fallen foul of their fair use policy and without warning restrict my broadband connection to whatever extent they wish and for however long they wish and still continue to bill me the same amount of money even though they neither specify the acceptable usage limits or provide me with any mechanism by which I can determine my current usage?

That doesn’t seem like fair use to me.

ORDB – Gone But Not Forgotten

ORDB has posed another Web traceability problem for me. They recently announced that they were closing down their Open Relay database that has been used by mail servers to help identify incoming mail likely to be considered spam based on knowledge of the mail senders IP address; the idea being that if enough people reported spam to a centralised organisation then a database could be maintained that could track IP addresses from which spam appeared to originate and then that organisation could provide a lookup service to mail servers which would allow them to rapidly check all incoming mail to determine if it originated from one of these blacklisted IP addresses.

The problem is that although I recall reading about their impending closure, there is now nowhere authoritative that I can find that confirms the current status; their website, at http://ordb.org/news/?id=38, where the original news item was posted (I believe) is currently offline and the word on numerous online sites is that the ORDB service is now reporting all IP addresses as being blacklisted in an attempt to force mail server administrators to remove the ORDB database from their list of blacklist services, presumably to reduce the load on their servers if they decide to re-open in some form. Looking through online forums and blog posts it is clear that plenty of people are experiencing the results of this change.

There is really no technical issue here for me – the correct solution is simply to remove ORDB from my mail servers, which is something I did a while ago anyway. The real issue is that when I googled for “ordb” it was apparent that most of the articles were based on the same sources at www.nabble.com and in turn Slashdot which reported the nabble.com message. The one site I did find that reported a conversation with a former ORDB operator was on a German site at www.heise.de which reported:

Andreas Plesner Jacobsen, a former ORDB operator, explained to heise online sister publication iX that this measure has been introduced because the zone is still swamped with queries. The intention is to get mail server operators to stop using ORDB. Simply deleting the domain was not a viable alternative, since the load would then merely be directed to the .org name server.

Ironically, it seems that the only way ORDB could get people to act to remove them from their mail servers was by breaking their service in a way which forced administrators to investigate – simply making an announcement on their web site was clearly not sufficient.

MusicBrainz

MusicBrainz is a community-driven music meta-database. It is almost a Wikipedia of the music world. Content is maintained by a community in which changes have to be approved by a vote before they become permanently accepted into the database and a team of dedicated moderators oversee the whole thing. The words orderly and consistent spring to mind; emphasis is placed on consistency of style (by that I mean, for example, capitalisation and abbreviation conventions) and correctness. In return for their input, community members are rewarded with a great tool for maintaining their own music databases – the tags in their own music collection. The tool is called Picard and is free to download. In acknowledgement of the quality of the database MusicBrainz has now been licenced by an impressive list of customers including MusicIP, The British Broadcasting Corporation (BBC) and Last.fm.

At some point during the Christmas holiday of 2005 I began the process of ripping my CD collection to allow it to be recalled and replayed in a more controlled manner and to reduce the amount of space that it occupied. I opted to make it visible on my home LAN using a combination of a Linksys NSLU2 (affectionately known as a SLUG) and a 250GB Buffalo DriveStation. The DriveStation is a USB hard drive and the Linksys bridges such a USB drive to an ethernet network. The Linksys will support two drives, although I’ve not needed to use both here.
The nice thing about the Linksys is that it runs Linux and can be customised to perform a number of task in addition to file serving. Both devices sit quietly tucked out of the way in an upstairs room and are directly connected to my wireless broadband router. The Linksys runs the ubiquitous (at least in the Unix world) Samba SMB file server which allows it to look just like a networked PC to other PCs on the home network. Files can be accessed from its drive(s), given the appropriate permission, as if they were on a PC but, of course, there is no fan noise or hot processor or display and so power consumption is at a minimum. The beauty of this arrangement is that they can be left running and so are always available.

I used Windows Media Player to rip the, roughly, 5,700 tracks from CD to mp3 format. Media Player makes a great attempt to tag the mp3 files correctly but for an irritatingly large number of tracks the information is either incorrect or inconsistent. This is where MusicBrainz comes to the rescue. Specifically, MusicBrainz Picard, their free and open source, cross-platform music file tagger.

Picard uses the MusicBrainz database to correctly and consistently tag mp3, wav, vorbis, flac, mpc, mp4 and wma format files. If asked to identify a music CD it will recognise the artist and release based on an analysis of the content of the CD which it uses to construct a unique disc-id which can be compared to known disc-ids in the MusicBrains database (at the time of writing there are approximately 228,000 such known disc-ids). Alternatively, Picard can recognise individual music files by a form of audio fingerprinting and makes a special effort to associate clusters of music files with a particular release or album. If neither of these techniques succeed the GUI allows manual associations to be made with the correct titles from the database.

Once associations have been made, Picard displays the tag information currently stored in the music file alongside the suggested information (from the MusicBrainz database) together with an indication of closeness of fit and allows (selective) correcting of the tags in the music file. Plugins to Picard allow you to pull down cover art of incorporate genre information from Last.fm but I haven’t tried either of these.

Reading Ian Dixon and Ed Bott’s postings on how they organise their music collections made me realise just how many different ways there are to achieve the same outcome. Where I think the MusicBrainz tools score is in the quality of the database that sits behind them.

BBC iPlayer Installation Woes and Success

I’ve seen several postings about issues people have had with BBC iPlayer installations and many more posts containing descriptions of attempts to solve these issues. In general, I found that installation on three machines (two Vista and one XP) in the house was very straightforward but did have a problem on one of the three machines, a Vista machine.

I believe my problem may have been caused by running the installation program as administrator but, whatever the cause, the symptoms were that I could download programs with the download manager but when I attempted to play any program by clicking on “Watch Now” the iPlayer would simply silently refuse to do anything. If I modified the iPlayer settings to allow me to use “my default media player” (in my case Windows Media Player) it would also fail to play the video and (if I recall correctly) display a message about being unable to open or find the media.

The solution in my case was entirely related to file and folder permissions and the following steps resolved everything:

1. Find the folder where the iPlayer stores the downloaded media files. In my case, C:\Users\Public\Videos\My Deliveries\iplayer_live
   This information may be determined from the iPlayer settings page.
2. Look at the permissions on one of the media files in the folder. The files will have names like 8780437_74568151_Selection_DOWNLOAD.wmv.
   Right click on a file and choose properties->security and check that your username (or everyone) has at least “read & execute” and “read” permission on the file.
   If it doesn’t then it is quite likely that this will be your problem and so continue with the steps below.
3. On a Vista machine for which the installation worked first time I find that the files have “read & execute” and “read” permission
   for my username and “full control”, “modify”, “read & execute”, “read”, and “write” permissions for my group (Administrators).
   You will need to manually add sufficient permissions to allow you to at least read the file. I think I took the heavy handed approach that
   we all take in times of desperation and applied “full control” to the .wmv and its accompanying .smi file (the file with the same name
   but different file suffix).
4. Test the file, by either right clicking the .wmv file and choosing “play”, or by clicking the corresponding “Watch Now” link
   in the iPlayer “My Downloads” window.
5. If the file doesn’t play (allow a suitable startup time) then this isn’t the solution to your problem.
6. If the file does play, then you will most likely need to perform a similar fix for all of your downloaded programs.

In my case, the real problem lay in the permissions associated with the folder C:\Users\Public\Videos\My Deliveries\iplayer_live.
My solution was to change the permissions on the folder to allow “full control” to “Everyone” and to ensure that such permissions were subsequently inherited by all new files that the iPlayer download manager created in the future.

Who Can You Trust?

The following two snippets of information were taken from Tom Flynn’s Did You Know? column, part of the Point of Enquiry podcast from the Center for Inquiry:

According to a June 2005 Gallup pole: 73% of Americans believe in the paranormal, 41% believe in ESP, 37% believe that houses can be haunted by ghosts, 25% believe in astrology and 42% believe that people can sometimes be possesed by the devil. According to a later Gallup pole: 24% of Americans believe that Alien beings have visited Earth. According to the Roper Center for Public Opinion at the University of Connecticut: 4 million Americans have reported that they have been abducted by aliens.

The second largest belief group in America is the non-religious. The total number of individuals who are not associated with a religious group in America is roughly 48 million, about 16% of the total population. They outnumber every other faith tradition except Christianity. This is more than the number of African Americans or Gays and Lesbians in America. Most of these unaffiliated (roughly 30 million people, which is about 10% of all Americans) are secular, humanist, atheist or agnostic but there is only one openly agnostic, aetheist or secular congressman or senator.

Both contain some interesting facts. But how do we know they are true? Tom names his sources, in this case Gallup and the Roper Center but that’s as far as a listener or a web page reader can really get towards verifying the sources without paying to see the actual surveys.

But this is nothing new. Whenever anyone tells us a fact we must make some judgement about the truthfulness of their statement and that is exactly what we do when reading a website or listening to a podcast. In my experience, the following weighs heavily when making such
judgements:

1. Who else endorses the podcast or web site?
2. Over a period of time what proportion of their productions remain within my own determined bounds of credibility?

On the first count, both Richard Dawkins and Martin Gardner have contributed and so have, in some sense, endorsed the organisation. Both men are respected academics in their respective fields of biology and mathematics and that is enough to satisfy me for the moment.

On the second count, only time will tell.

BBC Podcasts – December 2005

Earlier, I made reference to the published BBC podcast figures for July 2005. The BBC do not distinguish between on-demand streamed programmes and podcast downloads in their ongoing download and podcast trial. The figures for December 2005 are now available on the online statistics page and show that the monthly downloads figure for the BBC Radio 4 In Our Time programme has grown from 25,000 to 145,000 in the six months leading up to December 2005.

That’s quite a jump.

Top Ten Sources

I’ve just finished listening to Dave Winer’s Morning Coffee Notes interview with John Palfrey, founder of toptensources.com. In the interview, John Palfrey explains the ideas behind his site which empowers selected editors to manage a list of their top ten sources (a source being a web site or a blog or a feed). I like the sound of anything that helps improve my productivity whilst browsing, after all that is what is driving the aggregator movement. But in the back of my mind I kept thinking of how much I liked the Radio UserLand updates link which lets me see which other weblogs in the UserLand community have just been updated. Consider the following observations that I made whilst browsing the list of updated weblogs:

1. It would appear that most of the people who have gone to the trouble to subscribe to Radio UserLand are interested in maintaining reasonable content.
2. A certain amount of pleasure is derived from the uncertainty over the real quality of the weblog content.
3. I like the fact that on each visit the updates page contains a different list of weblogs.
4. A quick scan down the list of updated weblogs is all the human eye needs to identify interesting looking weblogs (in much the same way that it rejects spam emails by their titles).

Observations one and two combine to suggest that, within certain bounds, I am looking for an element of variability. Observations three and four hint that I would be happy to apply a personal filter to a larger list of candidate weblogs and that I enjoy the random nature of the presentation.

What does all this have to do with toptensources.com? Well, I question whether the top ten sources is what people really want or indeed whether there really is such a thing (even in the minds of the editors). For my part, I’d like to see a good ten random but reliable sources. How about tentopsources.com – nobody has taken it yet!

A COMDEX Footnote

Tidying out some papers this weekend I came across a copy of an editorial from BYTE magazine for February 1982 covering the November 1981 COMDEX exhibition in Las Vegas. Right at the foot of the article, in a section headed “Other Software Developments”, the editor notes:

Intel has signed agreements with both Microsoft and Digital
Research to distribute both companies’ operating systems for a wide
variety of Intel microcomputer systems and boards. This is a
continuation of an interesting phenomenon that began when IBM announced
it was going to make available both Microsoft’s DOS operating system
and CP/M-86 for the IBM Personal Computer. With corporate giants like
Intel giving Microsoft and Digital Research a boost, it appears that
both families of operating systems will coexist for quite some time.

Within three years the game was pretty much up for Digital Research’s CP/M with Microsoft’s MS-DOS gaining popularity. Digital Research almost came back again with DR-DOS but by then the goal posts had moved towards the GUI arena.

Thirty Years of Sharing

The January 2006 edition of Dr Dobbs journal runs an article entitled Dr Dobb’s Journal @ 30 in which Michael Swaine tells the history of the development of the magazine from its birth in early 1976 as Dr. Dobb’s Journal of Computer Calisthenics & Orthodontia to its current form. The magazine was conceived at the point in time where the prospect of building or owning a personal computer became a financial reality for many hobbyists and the excitement that accompanied it was easy to sample.

A number of hardware vendors began turning out microcomputer kits which consisted of:

• A simple CPU, such as Intel’s new microprocessor chip developed for the calculator market.
• A storage device which might be a fixed or floppy disk, a magnetic or paper tape or more commonly a cassette recorder.
• Some form of I/O device which, given a suitable interface, might be a teletype or even a keyboard and display but other times was little more than a set of dip switches and lights.

The concern among the hardware vendors was that people would not find an application for the hardware and so the marketers of the time declared that their use was “limited only by your imagination”.

In common with other electronic and computing magazines of the late 1970s and early 1980s, the journal retained a hands-on hobbyist feel even as more and more business application began to fill the non-editorial and classified pages. The emphasis was on sharing information to further extend the boundaries of what could be achieved on a limited budget – the journal even published a 4K BASIC language implementation in hex to key into the Altair.

In the 1980s, magazines commonly published listings as part of an article or in an appendix. When the computer games craze began, it was common to return from the newsagent with a fresh copy of a favourite computing magazine and begin the process of keying in pages of BASIC listing in the hope of playing what most modern teenagers today wouldn’t recognise as a game. Often the listing would contain a printing error and the following edition of the magazine would print the corrections required to make the game run.

Over the decades the journal published volumes of source code, ranging from byte-saving coding tricks for the 8080, Steve Wozniak’s floating-point routines for the 6502, Lawrence Livermore Labs BASIC, John Starkweather’s PILOT, implementations of PASCAL and FORTH, the full source code for an 8080 kernel, a portable screen-oriented editor and, very importantly, two C compilers and the beginnings of a toolset in C. All of these tools were freely available for the use of its readers.

Almost thirty years ago Jim Warren, the incumbent editor, wrote:

It is this open sharing that particularly delights me…We must all do what we can to encourage it. The sharing of ideas…allows us to stand on one another’s shoulders, instead of standing on one another’s feet…So continue to share your ideas, and continue to share your excitement.