RSS and Mail, Revisited

First off, good news for the blog purists: My RSS feed now picks up the first heading from each post and uses that as the item title. It turned out to be a fairly simple hack once I stumbled upon the right spot in the code, so there should be no more blog/foobar items, unless they happen to have no heading.

I spent some time yesterday fooling around with newspipe, and came up with some interesting conclusions.

(If you've never paid any attention to my Quest, newspipe is a way to transform RSS feeds into e-mail messages, and one of the approaches I've been looking into in order to tame the RSS hydra.)

First off, it's suicidal to run that sort of thing with my full OPML subscription list and pipe the resulting couple of thousand messages through SpamAssassin - my mail box topped out at a 360 loadaverage (probably the most excitment it's seen in a couple of years) and quickly ran out of file descriptors. But that's entirely my fault, and if I ever commit to using it this way I'll hack in standard mbox support and have newspipe append directly to a file.

Second (and this is the interesting bit), once I had worked around the usual MIME bugs to get inline images to show in Thunderbird and Mail.app (at this point it is broken in Thunderbird again and only works in Mail.app), I found that usability wasn't as good as I expected.

Now this is both Thunderbird and Mail.app's fault, and needs some careful consideration:

newspipe lets you join new posts from any given RSS feed into a single "digest" message, which is a great way to read somewhat terse but frequently updated feeds like /.'s - and also limits the amount of mail you're likely to get.

There are a couple of disadvantages, though. The first is message size: I got a whopping 3MB message with the Planet GNOME digest (partly due to Nat's great photos), which is fine if you're reading mail locally but a total pain if you're trying to keep up to date from a remote location (which is my typical scenario).

The second is that digest feeds impact your ability to forward interesting items on to friends and colleagues - you have to edit the thing to forward the interesting bits (there is also a very nasty bug in Mail.app that converts inlined JPEG images into TIFFs when you try to forward messages, but that's something I have to look into).

The third (which is a direct consequence of these two) is that you're entirely dependent on the MUA's capabilities. I can't make any sense of news items using my Pocket PC, for instance, even though it has a fairly decent IMAP client (newspipe renders items in an alternate text-only version, but not all MUAs take to it).

So the web-based approach wins this one (web-based RSS aggregators work across all platforms, even if they can't forward items).

Thunderbird has the extremely annoying default behaviour of taking up screen real estate with an "attachments" pane (instead of collapsing it as Mail.app does). This is fine for a couple of inline images, but suicidal for, say, forty (which is about what you'll get if you set up a digest feed from Engadget) - about half the preview pane is taken up by the attachments box, and there is no way to turn it off.

Scratch another one up for web-based aggregators: the layout is always the same, regardless of platform.

Not all feeds have valid e-mail addresses that newspipe can use as From:. This can be worked around by forcing a default e-mail address, so it's no biggie. But it impacts your ability to sort by feed, and since neither Thunderbird nor Mail.app have what I would call decent dynamic filtering support (i.e., like Outlook's ability to define on the fly views, queries, groupings on a specific header), you're stuck with sorting by Subject:, Date: (which is currently mangled) and little more.

Anyway, the bottom line is that I can't get consistent UI behaviour across platforms unless I set up Thunderbird everywhere, but I won't give up on Mail.app (I vastly prefer its clean looks and intuitive UI to Thunderbird's).

I can use the X-Feed-Something headers with procmail, so I can separate feeds by folder, but that creates another problem: my IMAP folder tree expands beyond any definition of humungous, and creating folders automatically based on a mail header field is a bad idea.

So, web-based aggregators win again, provided you can hack in whatever kind of view you like.

In both MUAs, using the space bar to page down inside a message and skip to the next unread message works - sort of. In Mail.app, behaviour tends to be somewhat inconsistent (depending on what your message sorting order is), with the focus shifting to the previous message, regardless of whether it's been read or not. This might be configurable, but I'm playing dumb and using the defaults, since a big point of using an RSS aggregator is making it both easy and efficient to read through whole swathes of news items.

NetNewsWire, of course, has this down pat. It's still one of the most efficient aggregators I've used, but the web-based approach wipes the floor with it too - space bar scrolling (on any browser) works faster.

So the score so far is:

• Web-based RSS aggregation: 4
• Mail-based RSS aggregation: 0

I'm not factoring second-tier usability (i.e., manipulating items, forwarding them, etc.) yet, mind you, but just reading stuff is way easier with a web-based aggregator (which is why I keep hacking at my Feed On Feeds install).

The other usability factors (as far as I'm concerned) are related to the ease you can perform actions such as marking items as read/unread/flagged for follow-up, filtering them, forwarding them, archiving, and performing some sort of (Bayesian?) training to highlight more relevant items. All of these might be simpler to do with MUAs, but the only MUA I know of that can perform all of this out of the box with a decent UI is... Outlook (sure, Evolution can do some of it, but it's not as good - yet).

Entourage 2004 is supposed to have a lot of Outlook-like features, but I don't intend to ever use it (unless my next corporate laptop happens to be a PowerBook, which is a slim chance at best).

So it's likely that the web-based aggregator will win again (all of what I mentioned is doable on a web interface), especially if I can get my XSLT act together and render feeds in both phone-friendly and PDA-friendly formats.

So, besides testing newspipe a bit more, I'm going to resume fiddling around with Feed On Feeds and see what comes of it.

Which, in the end, means the RSS hydra lives to fight me not just another day, but at least a good couple of months.