The Eleventh Commandment

After a week of nearly continuous problems with "transparent" proxies at the local incumbent's ISPs (and the inevitable distress calls of those who regard me as their "personal helpdesk staff"), the whole "transparent" proxy thing came back into (painfully) sharp focus.

Modern ISPs, ignoring the fact that traffic optimization and cacheing only saves bandwidth if you have a uniform user base (i.e., people who do in fact visit the same sites, and whose traffic falls inside the cache's "sweet spot"), are relying more and more on "transparent" HTTP proxies, foisted on them by suppliers who present absurd traffic saving claims, complete with flip-up "business cases" designed to fool pointy-haired bosses who don't even know the basics (warning: PowerPoint link).

Now, anyone with a basic grasp of probability theory will tell you that, as your customer base increases, the likelyhood of them visiting exactly the same sites does not follow in a linear fashion. In fact, the total amount of (unique) visited URLs will increase with a steeper gradient, and the cache's efficiency will soon degrade to a point where (and this is the bit your cache vendor didn't tell you about) it's essentially adding nothing but delay.

Period. You are now the proud owner of a six-rack packet delay unit with built-in disk storage (you might as well replace it with a cross-over 10Mbps Ethernet cable).

Oh, and in case you haven't noticed, any sort of "transparent" cache is the single most noticeable point of failure you might have on your network (it even tops your e-mail server).

Back in the days where I worked at "normal" ISPs (even though the distinction is blurring fast), we had a spoof of the Ten Commandments up on the wall that read somewhat like this:

 1. Thou Shalt have No Other Protocols besides TCP/IP.
 2. Thou Shalt Not Take the name of Ethernet in vain.
 3. Remember the Uptime, to keep it holy.
    Seven days a week shall you labor, and have no rest.
 4. Respect thy Uplinks and Standards, honor UUNET and the IETF.
 5. Thou Shalt Not Disconnect the Customer.
    Ever, no matter how much traffic he/she is generating.
 6. Thou Shalt Not Kill Spammers, but thou shalt
    not tolerate their presence on your network either.
 7. Thou Shalt Not Peer with the Competition.
 8. Thou Shalt Not Tweak your BGP Tables without calling your peers.
 9. Thou Shalt Not Covet thy Competitor's Fiber.
10. Thou Shalt Not Covet thy Competitor's IP address blocks.

And an eleventh (often unspoken) commandment:

11. Thou Shalt Not Fiddle With Thy Customer's Bits

Guess what - we didn't. But the battle to save cash by saving bandwidth (instead of buying more and lowering cost) is raging, and most broadband ISPs in Portugal now use "transparent" caches to try to curb their customer's insatiable craving for bandwidth. This despite the fact that HTTP traffic is being overcome by P2P traffic in more than a few instances, and that the purported "savings" are being offset by the declining cost of upstream bandwith at a three-to-one ratio.

It is now actually cheaper to buy more upstream bandwidth than to cache, and that is backed by the strong Euro trend - which most ISPs ignore when doing internal accounting (and even peering agreements).

But the real problems are subtly different. In an attempt to flog more revenue out of the downtrodden dial-up access market, some ISPs are turning to split-architecture optimizers. These are special clients which act as local proxies and/or redirectors and talk to a central proxy server, compressing traffic in the dial-up link by the (not so) subtle means of (among other things) degrading image quality.

These are being sold (or given away) as part of a new set of offerings from dial-up ISPs, and invariably install a ludicrously intrusive piece of software on your machine that basically compresses anything going in or out.

Mind you, blocking banner ads - which total 60-70% of bytes transferred as part of your average portal/news page these days - never seemed to cross their minds. And banners are, by design, the most caching-averse content on the Net, with every trick in the book (and some which aren't) being used to ensure the browser gets a freshly minted banner straight from the server.

(Doing the math on how much bandwidth that would save - and how much ad revenue ISPs who operate portals would lose - is left as an exercise to the reader.)

Still, the incumbent's cable ISP is now offering free anti-virus and firewall software (they must finally have noticed they were getting routinely blocked at upstream providers). That's a positive move.

Now, one of the folk I had the good (mis)fortune to help yesterday had precisely one such piece of garbage installed. It made it completely impossible to surf the Net when the proxy was down, and it made a complete hash of at least one Flash animation (I might put up a screenshot if I can be bothered to crop out the sensitive bits). Needless to say, I was not impressed.

Optimizers do have their uses, though, especially on mobile networks. Of course there is the odd problem with mangled images, but generally speaking it is a measurable improvement in user experience and saves the customer a serious amount of money. However, there are a few caveats - such as the onset of 3G services, which should render them obsolete due to the demand for higher-fidelity content.

The real issue, as always, is people. Modern ISPs, often lacking people with real Internet experience and know-how, have managed to break every single Commandment I presented above (often within record time spans), and the eleventh is no exception. While the consumer is likely to appreciate the benefit of moderately intrusive add-ons like Spam and anti-virus filtering in mail services, anything that messes with the gestalt of Web surfing (from image degradation to outright content corruption), despite likely to cause a ruckus, will eventually get deployed. More likely so if it purports to save cash.

If you're in Portugal and are having trouble with "normal" "transparent" proxies, picture what would happen if those became optimizers as well.

(What really worried me was the way these things appear to muck up Flash and the occasional JavaScript, which means they're not just ripping whitespace anymore.)

Now, "transparent" anythings are not really a problem. The real problem is not being able to opt out anyway (even if you're forced to sign something that states "yes, I understand I'll consume more bandwidth and pay more").

The fact that consumers are forced to use "transparent" proxies without any way to opt out (other than switching ISPs) is the best indication of how clueless both technical and marketing folk are at these ISPs.

And the saddest bit is that, this being Portugal, credible alternatives are thin on the ground (the incumbent controls most of the cable and ADSL broadband market), so people can't even use the only thing that makes them listen: switch to another ISP.

Here's to monopolies. Yay.