Still Catching Up

A week later, I still haven’t quite caught up on my personal backlog, and as such I decided to take the time to systematically go through my notes and to-dos and make an attempt at consolidating them. Being a “checklist person”, I tend to use them to plan ahead, but lacking the time to follow through at home, I ended up with around a dozen lists whose top item was “Sort photos”.

And so I did–or at least started to.

My personal filing system is still the same: a plain, unadorned filesystem tree, with uniform filenames courtesy of jhead and a few Automator workflows:

Photos-+-2001-+-01-+-200101012230.jpg
       |      |    .
       |      |    +-200101311200.3gp
       .      |
       .      +-02-+-200102010830.jpg
       .      .    .

However, things have become rather complex over the years:

• I now have to deal with multiple sources: iCloud, OneDrive camera rolls, and the odd SD card. I haven’t plugged in a camera to any Mac for years.
• Besides photos and several flavours of videos (most converted to MPEG-4), I now have an increasing number (and kind) of RAW formats, from DNG to CR2.
• Reading metadata off all of those (and managing it) has become somewhat of a puzzle.
• The current standard for sidecar metadata, XMP, is the ugliest piece of XML-insipred trash I’ve ever had the displeasure of dealing with.

…and, regrettably, darktable insists on creating XMP files all over the place, which somewhat complicates matters whenever I need to move or rename files.

But I’m learning to live with this (and have been adapting my workflow and scripts to cope), with the result that I have started making some headway into my photo backlog, which is now somewhere in mid-2017.

As always, shootShifter is invaluable whenever I come across events shot with multiple cameras and need to line everything up to the second, but it helps if I split photos across folders on a per-camera basis, so I revamped one of my old helper scripts:

#!/bin/env python3

import os, re, sys, subprocess, shutil

db_pattern = re.compile('^\d{4}-\d{2}-\d{2} \d{2}\.\d{2}\.\d{2}')
jhead_pattern = re.compile('^\d{14}')

files = os.listdir('.')
for f in files:
    name, ext = os.path.splitext(f) 
    if db_pattern.match(name):
        os.rename(f, name.replace(' ','').replace('-','').replace('.','') + ext)

photos = filter(lambda x: '.jpg' in x.lower(),os.listdir('.'))
for name in photos:
    pipe = subprocess.Popen("jhead %s" % name, shell=True, stdout=subprocess.PIPE).stdout
    # yes, this is somewhat LISPy, but it saves me a lot of time:
    meta = {k: v for k, v in list(map(lambda x: map(lambda x: str(x.strip()), x.split(':',1)),map(lambda x: x.decode('utf-8').strip(),pipe.readlines())))[:-1]}
    try:
        print(meta)
        folder = meta['Camera model']
        name, ext = os.path.splitext(meta['File name'])
        if jhead_pattern.match(name):
            # split by year, month, day, camera (makes it easier to process dupes)
            folder = '%s/%s/%s/%s' % (name[0:4],name[4:6],name[6:8],folder)
        if not os.path.exists(folder):
            os.makedirs(folder)
        shutil.move(meta['File name'], folder)
    except Exception as e:
        try:
            print("Error %s while handling %s %s" % (e,folder,name+ext))
        except:
            print("Got %s" % e)
        pass

The above uses jhead to split files according to date and camera model. It only deals with JPEG files, but I have a very similar (and shorter) one with mdls that does a similar thing for all image formats macOS understands by looking for kMDItemAcquisitionModel, but this one is likely to be of more general interest.

Of course, regardless of how much automation you use for photography you still have to eyeball things, and my new iMac, thanks to its glorious 5K display and the 4K panel I have alongside, is hands down the best machine I’ve ever owned for photo work, so things are going fairly smoothly.

There are only two caveats:

• No amount of CPU power will ever be enough to cope with RAW satisfactorily in my book
• darktable can be dog slow when navigating large collections

The latter is not due to formats or any real processing, but seems to be a rather unfortunate side effect of the GTK layer it’s built upon and the current state of OpenCL support on the Mac.

It is so obviously an UI lag problem that I’m quite surprised it hasn’t been fixed yet. But most darktable users seem to be using Linux, and as such I have a feeling macOS support isn’t a high priority for the project (something that this issue and this one seem to confirm, seeing as they’re still extant and have had little to no attention).

Moving on to stuff I can control, I expect to spend some time addressing the lack of a photo gallery in this site fairly soon. I’ve been meaning to move off Flickr for years, and I’ve been doing some fairly neat OpenCV and neural network trickery, so there are a few things I want to try.