Still Catching Up

A week later, I still haven’t quite caught up on my personal backlog, and as such I decided to take the time to systematically go through my notes and to-dos and make an attempt at consolidating them. Being a “checklist person”, I tend to use them to plan ahead, but lacking the time to follow through at home, I ended up with around a dozen lists whose top item was “Sort photos”.

And so I did–or at least started to.

My personal filing system is still the same: a plain, unadorned filesystem tree, with uniform filenames courtesy of jhead and a few Automator workflows:

       |      |    .
       |      |    +-200101311200.3gp
       .      |
       .      +-02-+-200102010830.jpg
       .      .    .

However, things have become rather complex over the years:

  • I now have to deal with multiple sources: iCloud, OneDrive camera rolls, and the odd SD card. I haven’t plugged in a camera to any Mac for years.
  • Besides photos and several flavours of videos (most converted to MPEG-4), I now have an increasing number (and kind) of RAW formats, from DNG to CR2.
  • Reading metadata off all of those (and managing it) has become somewhat of a puzzle.
  • The current standard for sidecar metadata, XMP, is the ugliest piece of XML-insipred trash I’ve ever had the displeasure of dealing with.

…and, regrettably, darktable insists on creating XMP files all over the place, which somewhat complicates matters whenever I need to move or rename files.

But I’m learning to live with this (and have been adapting my workflow and scripts to cope), with the result that I have started making some headway into my photo backlog, which is now somewhere in mid-2017.

As always, shootShifter is invaluable whenever I come across events shot with multiple cameras and need to line everything up to the second, but it helps if I split photos across folders on a per-camera basis, so I revamped one of my old helper scripts:

#!/bin/env python3

import os, re, sys, subprocess, shutil

db_pattern = re.compile('^\d{4}-\d{2}-\d{2} \d{2}\.\d{2}\.\d{2}')
jhead_pattern = re.compile('^\d{14}')

files = os.listdir('.')
for f in files:
    name, ext = os.path.splitext(f) 
    if db_pattern.match(name):
        os.rename(f, name.replace(' ','').replace('-','').replace('.','') + ext)

photos = filter(lambda x: '.jpg' in x.lower(),os.listdir('.'))
for name in photos:
    pipe = subprocess.Popen("jhead %s" % name, shell=True, stdout=subprocess.PIPE).stdout
    # yes, this is somewhat LISPy, but it saves me a lot of time:
    meta = {k: v for k, v in list(map(lambda x: map(lambda x: str(x.strip()), x.split(':',1)),map(lambda x: x.decode('utf-8').strip(),pipe.readlines())))[:-1]}
        folder = meta['Camera model']
        name, ext = os.path.splitext(meta['File name'])
        if jhead_pattern.match(name):
            # split by year, month, day, camera (makes it easier to process dupes)
            folder = '%s/%s/%s/%s' % (name[0:4],name[4:6],name[6:8],folder)
        if not os.path.exists(folder):
        shutil.move(meta['File name'], folder)
    except Exception as e:
            print("Error %s while handling %s %s" % (e,folder,name+ext))
            print("Got %s" % e)

The above uses jhead to split files according to date and camera model. It only deals with JPEG files, but I have a very similar (and shorter) one with mdls that does a similar thing for all image formats macOS understands by looking for kMDItemAcquisitionModel, but this one is likely to be of more general interest.

Of course, regardless of how much automation you use for photography you still have to eyeball things, and my new iMac, thanks to its glorious 5K display and the 4K panel I have alongside, is hands down the best machine I’ve ever owned for photo work, so things are going fairly smoothly.

There are only two caveats:

  • No amount of CPU power will ever be enough to cope with RAW satisfactorily in my book
  • darktable can be dog slow when navigating large collections

The latter is not due to formats or any real processing, but seems to be a rather unfortunate side effect of the GTK layer it’s built upon and the current state of OpenCL support on the Mac.

It is so obviously an UI lag problem that I’m quite surprised it hasn’t been fixed yet. But most darktable users seem to be using Linux, and as such I have a feeling macOS support isn’t a high priority for the project (something that this issue and this one seem to confirm, seeing as they’re still extant and have had little to no attention).

Moving on to stuff I can control, I expect to spend some time addressing the lack of a photo gallery in this site fairly soon. I’ve been meaning to move off Flickr for years, and I’ve been doing some fairly neat OpenCV and neural network trickery, so there are a few things I want to try.


No, this isn’t a Surface joke. In fact, that computer is alive and well, and has been an excellent counterpoint to my iMac over the past couple of weeks, during which I was working throughout the clock on a customer engagement. Looking back, it was a lot of fun, but two weeks of sleeping less than 6 hours a day tend to drain the wits out of you, so I’m still recovering.


Back to the iMac

Last weekend, I got a new 27” iMac. Despite my earlier musings towards going fully thin-client and getting dual 4K displays, I was so fundamentally fed up with my current office setup (a Mac mini with dual displays, which lasted me all of eight years) that I had to do something about it. Worryingly, and despite my ingrained distaste for all-in-ones after owning a few of the early models, the logic behind it was a no-brainer.


My Quest for Home Automation, Part 1

As the cold season started creeping in and the need to balance energy savings while keeping the house warm became more important, I began laying out plans for doing home automation “the modern way”, i.e., without clockwork timers all over the place and with some intelligence thrown in.