A simple matter of…

Surfing the web can be slow (I’m not bitter about being 1/2 mile from the [http://www.objectbase.co.uk/basingstoke/chineham/fttc/ rollout] of [http://en.wikipedia.org/wiki/Fiber_to_the_x Fibre To The Cabinet] in Basingstoke, oh no).

There are various tricks currently in use:
* Expiry headers which say that data will be valid until a certain time has elapsed. This is great for things such as images or CSS which rarely changes
* Caches (both in the browser and in the network) can also help
* [http://en.wikipedia.org/wiki/HTTP_ETag ETags] can check to see if content has changed or not
* Compressing data using gzip

I wonder if anyone has thought about using something like [http://en.wikipedia.org/wiki/Rsync rsync] instead. For example, even if the BBC news page changes, it’s likely that only 5% of it changes. As usual there is a trade-off here between CPU and bandwidth, however perhaps you could target specific versions – for example the previous version? Most laptop and desktop hard disks are so colossal these days that storing even gigabytes of cached data shouldn’t be an issue (although I do remember Netscape’s cache actually being slower than the network at times!)

It looks like there have been some attempts on this, mostly abandoned due to “too much bandwidth” – I wonder if that’s actually true? Reducing bills and latency to mobile phones for example.

* [http://zsync.moria.org.uk/paper/ Zsync] – client side rsync algorithm to reduce server load
* [http://rproxy.samba.org/ rproxy] – abandoned in 2002