Burning the candle at both ends

So I’ve been busy.

That’s about it, really. Work has been… interesting, if altogether too “dynamic” for me to be able to finish a few things, and I’ve been doing a fair share of coding in the evenings, mostly in Python and mostly using a mix of gevent and multiprocessing¹ to speed up things.

And yeah, that joke about using threads to solve one problem and having problems two is spot on. Today’s multiple-core CPUs are often under-utilised, but coordinating parallel tasks is hard, especially when you’re trying to do it within a tight budget (i.e., no fancy architectures, no overwrought solutions, and no time, either).

Still, I’ve learnt at least two useful tricks this week:

1. Using gevent inside multiprocessing can speed up heavily network-dependent tasks (at least simple ones, like checking links) by a factor of eight or more (gevent on its own isn’t too shabby, but using multiple cores works far better when you can switch contexts while waiting for network calls)
2. SQLite can perform beautifully with multiple writers if you issue a PRAGMA journal_mode=WAL to turn on write-ahead logging².

By all means get a “real” database if you must, but right now I can run 128 parallel RSS processing tasks atop SQLite without any serious issues (although the dataset’s highest cardinality is in the low tens of thousands), and that saves me the time and bother to install Postgres.

Still on the topic of speed, PyPy Beta 2 is out, and if you use brew my pull request was just accepted and you can install in on your Mac by using brew install --devel pypy.

It’s wicked fast, and even though it locks up on some of my multiprocessing code for reasons I cannot yet fathom (most likely database locking), it runs this site just fine and text indexing seems to be at least three times faster – your mileage may obviously vary.

Finally, and still on the Python front, I’ve been tinkering with Brython a fair bit – not as much as I’d like to, but enough to think that it’s going places.

Oh, and I’ve been doing Android x86 builds again. Silly old me – it’s like a bad habit you can’t break.

All of this has taken a toll, though (I’ve been sleeping something like six hours a night on average), so I’m going to have to dial it back a bit for a few days – but it’s fun, and considering that for the first time in years we’ve had to dip into our savings, all of this work (real and hobby) was a welcome (and cheap) distraction.

Yeah, the economy is still going down the drain, and I’m trying to avoid thinking about money again. So please stop e-mailing me telling me how good it is in your country.

Let’s see how long I can keep up my good spirits, shall we?