So I’ve been busy.
That’s about it, really. Work has been… interesting, if altogether too “dynamic” for me to be able to finish a few things, and I’ve been doing a fair share of coding in the evenings, mostly in Python and mostly using a mix of
multiprocessing1 to speed up things.
And yeah, that joke about using threads to solve one problem and having problems two is spot on. Today’s multiple-core CPUs are often under-utilised, but coordinating parallel tasks is hard, especially when you’re trying to do it within a tight budget (i.e., no fancy architectures, no overwrought solutions, and no time, either).
Still, I’ve learnt at least two useful tricks this week:
multiprocessingcan speed up heavily network-dependent tasks (at least simple ones, like checking links) by a factor of eight or more (
geventon its own isn’t too shabby, but using multiple cores works far better when you can switch contexts while waiting for network calls)
- SQLite can perform beautifully with multiple writers if you issue a
PRAGMA journal_mode=WALto turn on write-ahead logging2.
By all means get a “real” database if you must, but right now I can run 128 parallel RSS processing tasks atop SQLite without any serious issues (although the dataset’s highest cardinality is in the low tens of thousands), and that saves me the time and bother to install Postgres.
It’s wicked fast, and even though it locks up on some of my
multiprocessing code for reasons I cannot yet fathom (most likely database locking), it runs this site just fine and text indexing seems to be at least three times faster – your mileage may obviously vary.
Oh, and I’ve been doing Android x86 builds again. Silly old me – it’s like a bad habit you can’t break.
All of this has taken a toll, though (I’ve been sleeping something like six hours a night on average), so I’m going to have to dial it back a bit for a few days – but it’s fun, and considering that for the first time in years we’ve had to dip into our savings, all of this work (real and hobby) was a welcome (and cheap) distraction.
Yeah, the economy is still going down the drain, and I’m trying to avoid thinking about money again. So please stop e-mailing me telling me how good it is in your country.
Let’s see how long I can keep up my good spirits, shall we?
Although to be fair I’ve been looking for alternatives to solve at least one particular issue, and I’ve been slowly gravitating towards Java because the JVM is wicked fast these days and I have to live with the language anyway. The only think keeping me away are the shoddy build tools. ↩︎
Just make sure you switch off any file syncing software you might be using while running this, because it’ll go nuts with all the filesystem notifications. ↩︎