Avoiding the Chasm

April 17, 2008

Broadband Network Usage Monitoring and Measurement Tools

Filed under: Technology — Tags: , , , , , , , , , , — vextasy @ 8:10 pm

If you have found that your ISP has been restricting your broadband bandwidth the obvious question you will ask your ISP is why? If you ask that question the answer you are likely to get is that you have been using the “broadband service inappropriately”. You might also be told that you have exceeded your usage allowance but if you, like me, are on an unlimited contract you are unlikely to be told what the upper usage limit is. The reason you won’t be told the actual value of the upper limit is that it is likely to be more complicated than a simple figure and the reason that this is the case is because the ISPs are principally interested in avoiding router congestion at peak times. At off-peak times, such as in the early hours of the morning and during the working day, it makes little difference to an ISP if the capacity of their network is 25%, 50% or 75% used, they still have the same equipment costs and other overheads. But once the network reaches capacity, and routers are forced to drop packets, then customers start to notice and the ISP begins to get a bad name. For this reason most ISPs have, quietly, begun to apply traffic shaping at peak times.

Traffic shaping involves restricting the bandwidth of `heavy usage’ customers in such a way as to prevent them from interfering with the network experience enjoyed by `lighter usage’ customers. Unfortunately it looks like we are in for a lot more of this as network demand grows. The popularity of the Internet as a medium for watching media has rocketed in recent months and looks to continue to grow as more and more people switch their viewing habits from the more traditional broadcast medium to Internet based technologies. In the UK, the BBC iPlayer alone has been responsible for tremendous changes in network usage.

I recently found myself in the position of having determined from my ISP that I was being traffic shaped. Unfortunately, my router provided me with little help in identifying the volumes of data that were passing through my broadband connection each month but my ISP furnished me with figures which seemed to be considerably higher than I would have expected. Finding myself in a very weak position I decided to rearrange my home network to allow me to gain a better understanding of my broadband usage.

My broadband is supplied by BT on an unlimited tariff and I use the BT supplied broadband router (2-Wire 1800) which hosts two wired Ethernet connections, one to a PC and one to a network attached storage device, and a number of wireless connections to PCs or laptops. There are nine devices in total but typically a maximum of 4 might be actively using the broadband line at any one time, and by active I usually mean browsing web pages. Living reasonably close to our exchange we can manage to achieve a download speed of about 6.3 Mb/sec and even with the bandwidth restriction we still reach this speed outside of peak times.

WallWatcher

WallWatcher

The problem with having a wired network is that the only device that can really determine how much traffic is flowing is the broadband router as everything else talks directly to it and it funnels data into and out of the ADSL line. I attempted to get my existing router to log traffic information to a PC so that I could take a close look at what was travelling up and down the external line.

There are many different logging analysers available but the one I chose to use was WallWatcher a free tool with support for a large number of routers.

Tomato

Unfortunately I found that I couldn’t get WallWatcher to correctly recognise the format of the packet logs coming from my model of 2-Wire router. Linksys_WRT54GLIn my case the solution was to make use of a spare wireless router that I had which was not being usefully employed, a Linksys WRT54GL. This variant of the WRT54 family of router runs Linux and can be easily upgraded to run an alternative piece of firmware. I wanted to concentrate my network devices on the Linksys router and then run a single connection from the Linksys router to an Ethernet port on the back of the 2-Wire broadband router. I also wanted to make use of the fine bandwidth reporting available from the Tomato firmware which this router can be upgraded to run. The process of upgrading the router took about 10 minutes as it can be done from a menu option within the native Linksys firmware. Once Tomato was up and running on the Linksys it was easy to configure it to provide a good quality wireless network that replaced the old 2-Wire network and as an added benefit was also faster.

I configured the Linksys to store its bandwidth logs on a network Windows share and to forward its packet logs to WallWatcher as before and the results bandwidth-24hrswere immediately interesting. Tomato hosts a number of web pages that show bandwidth over varying periods of time. I could see straight away that the download bandwidth on the WAN port was considerably higher than I would have expected and at quite a sustained value (see the image on the right). The graph shows consistently high volume of download overnight and then a period of very low activity in the morning when all of the PCs were switched off followed by high usage again from about 2pm when they were restarted. Tomato also hosts a real time bandwidth display. Using these displays combined with WallWatcher it was easy to identify the PC responsible for the heavy usage and by examining the addresses of the remote end-points shown by WallWatcher it was also easy to determine the offending program.

TV Tonictvtonic_realtime

In my case, the program generating the traffic was the TV Tonic RSS service, a program responsible for downloading video podcasts from the Internet. I hadn’t realised that the program was still active as I had not made use of the client for a number of weeks. Incidentally, the TV Tonic client runs as an add-on to Windows Media Center (under Vista) and is quite a nice addition. Had I looked a little more closely at its configuration I would have noticed that not only did it have an option to limit the download bandwidth but also it had a download scheduler to control the time of day that it be allowed to download at all (both of these options would be nice to see adopted by the BBC iPlayer). I’m not quite sure why TV Tonic was downloading such large amounts of data but, not wishing to experience another month of bandwidth restriction, I immediately disabled the TV Tonic service and the Tomato real time monitor showed the corresponding reduction in bandwidth usage.

BBC iPlayerwallreviewer-out

Even after disabling the TV Tonic RSS service there still seemed to be a lot of network activity from my PC although I wasn’t running any obvious client program. A closer look at the WallWatcher log display showed a large number of incoming and outgoing UDP packets wallwatcher-iplayerbeing sent to external machines. WallWatcher comes with a charting tool called WallReviewer which gives a useful interactive graphical picture of incoming and outgoing traffic information over a given period of time. The WallReviewer chart of “Outbound Leaks by Remote Names” showed a large number of packets being sent to the machines iplaykdms82.telhc.bbc.co.uk and iplaykdms6.telhc.bbc.co.uk. The names of these remote sites suggested the BBC iPlayer might be responsible but the application wasn’t running and the option “allow programmes to be shared when you exit download manager” was not ticked in the iPlayer configuration dialogue so I had assumed that there ought to be no networking activity from the iPlayer Kontiki-based software. I found that if I disabled the Windows service named “KService” (which runs the BBC iPlayer program “C:\Program Files\Kontiki\KService.exe”) then all of this network activity stopped immediately. From the WallWatcher display it was clear to see that these packets were being sent about every 2 to 4 seconds but WallWatcher is not able to give any indication of the size of the packets.

Wireshark

To get a better indication of packet sizes a protocol analyzer is required. The “old faithful” in this area used to be called Ethereal but development on Ethereal has now been moved to Wireshark. Wireshark is simple to install and can be used on many platforms. It is also free and licensed under the GNU General Public License. There is a lot more to Wireshark that the casual user is ever likely to need and a basic knowledge of networking protocols and terminology helps but there is plenty of documentation.wireshark-iplayer

Running Wireshark on my PC confirmed that data packets were being sent to the BBC domain every two to four seconds but also showed that the packet sizes were small, 16 bytes of payload which by the time they have been wrapped in the UDP and IP packets amount to a 58 byte Ethernet frame. I find that having disabled the KService service I am unable to start up the BBC iPlayer but as soon as I re-enable the service the iPlayer functions as normal.

Conclusion

Having made these networking changes I am now in a much better position to know exactly how much traffic is being downloaded (or uploaded) over my broadband line and also able to detect this traffic early on to avoid triggering any ISP penalties. The tools required to monitor bandwidth are not expensive (in fact they are free) and are easily configured. I think that my ISP should have been able to give me the information that I needed to monitor and control my bandwidth – it feels a little like having been sold a car which has no fuel gauge.

One lesson that can be learnt from all of this is that it is becoming more and more important for anyone with a reasonable grasp of networking to take matters into their own hands to monitor their own network usage. I don’t see the ISPs relaxing their grip on our usage patterns in the short term, at least not until their own issues of congestion have been addressed. So by tightening up on wasted bandwidth we should be left with more to do the things that we really want to use it for.

About these ads

2 Comments »

  1. Hi,
    I have a question on wallreviwer if you have a moment please. I have DD-WRT logging outbound traffic to local PC and all looks good in wall watcher. When I open reviewer the only report I seem to get is ‘most active local ports’ which does show data.

    Changing the display/analysis options does not seem to change the chart but does reduce the number of records to 0.

    How did you get the chart above to display in reviewer please? Ideally I want to report based on local IP address to see who is doing what.

    Thanks/Regards,
    Ross

    Comment by Ross — May 14, 2012 @ 4:03 am

    • Ross,

      I’d love to help but it was four years ago that I wrote that post and I have moved on to a different system now. Actually, I pay Virgin Media about half of what I was paying to BT to get 103Mbit/s download speeds and a sensible fair usage policy. I use a modified Linksys router running the ‘tomato’ (open source) operating system to monitor my usage – but it isn’t an issue these days anyway. So sorry not to be of any help but I honestly can’t remember that software now.

      Comment by vextasy — May 14, 2012 @ 5:57 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: