Rui Carmo
September 23^rd 2018, at late afternoon · 10 min read

My Quest for Home Automation, Part 3

I think an update on my HomeKit setup is in order, especially given that the last time I wrote about it at length was back in March, and a lot has happened since then.

My primary goal was to automate heating, and once I got an Apple TV 4K to act as a home hub and set up some basic automation everyone was pretty happy with the results, since we could step into a warm kitchen and office in the mornings, with heaters turned on (and off) around the house to match our typical schedules.

I still don’t have any decent way of monitoring power consumption (I’ve been pushing that back because it requires spending too much time fiddling with ways to point LEDs at the ancient rotating disk meters we have). But the results were worth it, with only three caveats:

Some initial instability that appeared to go away once I upgraded to newer Sonoff-Tasmota firmware and changed the default settings to WifiConfig 4.
One socket tended to lose connectivity now and then (which I ascribed to coverage and later to the default firmware config, but never really figured out).
Our cleaning lady tended to unplug things at random (both sockets and my Apple TV), which played havoc with some of the scheduling (you should always account for human factors).

As the cold season waned I disconnected most of the Sonoff S20 sockets and put them in storage for next year, but two are still in operation (one to switch on a foot lamp a little before sunset and another as a Wi-Fi canary/testing device in my office). I’ve had zero real issues with them for over six months¹.

Beyond Heating

Besides that one lamp, I didn’t tackle any lighting automation (which apparently surprises a lot of people). We have enough natural light that it just isn’t something interesting to do, even though I did wire up a WS2812 RGB LED ring to a WeMos D1 clone² to build a prototype mood light–which was fun, just not that interesting or useful.

My next stated target was dealing with the air conditioners, but over Summer I didn’t find a good way to completely automate them, partly because it’s trickier than I thought from an IR perspective (the devices sit under windowsills, and the IR blaster I have relies in ceiling reflections and simply can’t pump out enough power to compensate for the outside lighting), and partly because Summer is effectively still on, but not to the point where they were needed.

We did, however, set up timed shutdown via their arcane remotes for the few times we used them, and I captured some of the IR hex codes for later use.

The Tech

I’m running everything as Docker containers on my trusty old ODROID U2, which is around five years old but still better than a Raspberry Pi 3 (it has twice the RAM and EMMC storage, which makes it extremely reliable): I have an MQTT broker, HomeBridge and Node-RED containers talking to each other, and my own device shims running as user-level daemons using Piku:

A lot of people keep asking me if I tried HomeAssistant, and I did. But since I want to stick to managing things from iOS devices and don’t really need the added complexity I find it largely redundant.

I would probably have ditched Node-RED in the meantime (since I simply cannot get its HomeKit accessory support to work reliably), but it’s turned out to be useful for testing MQTT-related stuff and desigining simple dashboards (although to be fair it’s a pain to do anything data-related on it).

One of its saving graces is that I can actually edit flows and dashboards from my iPad (it pops up radial menus if you use a touch-enable device to access the Web UI), so I intend to keep using it as a front-end. I’d just like to have better ways to deal with data and nicer charts, since the default ones are somewhat utilitarian:

However, the thing that I would very much like to replace is HomeBridge. I’m not overly fond of it and its Node underpinnings, so I’ve been investigating Python (HAP-Python) and Go (HomeControl) alternatives with some degree of success–a few weeks back I built a Python accessory that is much better at dealing with my dumb TV/set-top-box combo via IR, and included a presence sensor shim that notifies me whenever the kids arrive home and turn on the TV (by detecting the Chromecast plugged into it³).

Hey Siri, is the kitchen on fire yet?

Once configured as part of a second HomeKit bridge, those shim accessories work well enough with Siri and the IR blaster to switch inputs, pause live TV and turn everything off (which are the things I most need to do), and I suspect I will be expanding them with Siri Shortcuts on iOS 12.

So, in a nutshell, I’m effectively running two HomeKit bridges: HomeBridge (with a plugin that talks MQTT directly to the Sonoff sockets), and my own Python bridge, which deals with all the UPNP and IR stuff required to control the media center.

And the base experience was just fine with iOS 11: even without Shortcuts, as long as you give proper names to your devices and rooms, Siri is perfectly capable of dealing with “switch off everything in the living room”, or answer queries like “is the office window open?”.

Yes, it's actually clever enough to do this.

I keep leaving the window open and letting in mosquitoes, so a notification to go back and shut it after dinner is forthcoming–since Shortcuts can’t schedule things and the HomeKit app can’t do conditional alerts, I’m going to build another shim that will alert me and trigger a notification if the window is left open before bedtime.

Adding Zigbee Sensors

Although automation hasn’t yielded significant benefits during the warm season, I decided to start taking a serious look at adding more sensors. The main reason is that my ESP8266 temperature sensor works OK but is hardly nice to look at, with the added annoyance that battery life is still on the short side (I’m tracking this GitHub issue in the ESP8266/Arduino project, which will eventually provide a fix, but it’s taking too long).

Thanks to a friend’s recommendation (hi Paulo!) I decided to start looking at the Xiaomi Aqara product range, which uses Zigbee–something I’d shied away from due to compatibility hassles but that is perfect for low-power, long-lasting sensors.

A bit more digging led me to the zigbee2mqtt project, which has a rather impressive compatibility list and allowed me to build my own gateway by using an USB dongle with the CC2531 chipset (currently €5), which has the nice bonus of isolating IoT devices from my Wi-Fi network.

The Xiaomi gateway is HomeKit compatible, but I wasn’t interested in that–the sensors, though, were another matter, since they are rather nicely turned out pieces of hardware and some of them can be gotten off AliExpress by around €10 including shipping:

Aqara sensors, powered by 3V button cells.
Clockwise from bottom left: contact sensor (the tiny blob is the magnet), vibration, temperature, and an SD card for scale
(a banana would be too big).

So I ordered some Aqara gear (a couple of contact sensors, a couple of temperature sensors and a couple of vibration sensors) along with the dongle and TI flashing cables, and spent this weekend setting things up. I started by flashing the dongle and figuring out a way to encase the naked circuit board, since it is not something I want to leave unprotected. Fortunately, there is a readily available 3D printable cover for it (which has a decorative hole that lets in dust, but that’s fixable with a sticky label):

The CC2531 dongle I used, snugly ensconced in its newly minted PLA case.

While that was printing, I built my own Docker image for the gateway (there is an official zigbee2mqtt container image for ARM and Intel, but I like having more control), mapped /dev/ttyACM0 into the running container (which required some fiddling, but nothing special), and set up the devices using this nice Node-RED web dashboard:

Integration

The contact sensors were trivial to add to my Python gateway, and I’m currently in the process of doing the same for the temperature ones. But to have immediate feedback and a better feel for when they reported link and battery status (which isn’t done on every update) I simply whipped up a temporary MQTT logger in Node-RED:

This feeds into the nice little dashboard shown at the top of this piece, which is mobile-ready and did the trick until I finished wiring up the devices to HomeKit.

Caveats

Software-wise, everything is peachy. I have less than 500 lines of my own code to do everything I want, but even then I’m keen to find the time to rip out HomeBridge and my Python shims and redo the whole thing in Go, because even though things work and are quite easy to code, I would like to make it all more efficient and eventually move all the gateways to a Raspberry Pi Zero.

As to why code at all, I actually prefer writing code for sensor correlations rather than try to wire things up in Node-RED, where any kind of complex logic invariably deteriorates into my swearing at Undefined values and trying to debug stuff in a browser.

Hardware-wise, so far only the vibration sensor has given me trouble–partly because it’s harder to pair with the Zigbee gateway for some reason, and partly because it’s not fully supported yet. Moving it around and even walking past it generates some interesting debug log entries in the gateway, but I’m happy to wait until it’s fully debugged and merged.

But most importantly, I’m still figuring out Zigbee radio coverage and range limitations.

The farthest sensor updates less frequently than the nearest, and even though it is amazing that it even works at all (our walls are a mix of brick and concrete, and the flat partially wraps around an elevator shaft and adjoining pillars), it does work.

But I don’t like the signal levels in more extreme locations around the house and don’t want to relocate the current gateway, so I’m currently considering ordering a couple more CC2531 dongles (which can be reflashed as Zigbee routers and stuck in vacant USB ports) and seeing if I can get a mesh going.

I’ll also obviously be adding a few more contact and temperature sensors, and possibly experimenting with other Zigbee gadgetry.

Opportunities for Extra Fun

What I’m still looking for, though, are two simple things: The first is a doorbell sensor–I don’t want to wire anything up to the actual doorbell, but since it’s a metal chime I was expecting the contact sensor (which is essentially magnetic) or the vibration sensor to be able to detect when it rings. Alas, partially disassembling the doorbell to squeeze the sensors in closer to the chime didn’t yield any useful results.

I suspect I’ll have to build my own, or hack a contact sensor to get the magnetic reed deeper inside the doorbell mechanism.

The second is an MQTT-triggered buzzer/alert light to act as a doorbell extender, but that’s almost trivial, and I intend to re-purpose my RGB LED gadget for that once I can actually detect doorbell rings. Or, if I’m feeling lazy, I’ll just pop down to IKEA and grab some TRÅDFRI gear.

I just have to remember to not go there during one of their discount weekends (I value my sanity too much to be caught in one of those rushes).

Whenever the foot lamp fails to turn on, for instance, it almost invariably is due to something wrong with HomeBridge or my Apple TV having been unplugged during dusting. ↩︎
You can do it with Tasmota firmware and map it to HomeKit just fine, and I will probably build a bigger one once I find a decent frosted glass enclosure. ↩︎
I’ve put up the code for both Chromecast and Mediaroom detection on this gist in case anyone is interested. I decided against using asyncio this time because UDP handling is… just weird. ↩︎

Rui Carmo
September 18^th 2018, at breakfast · 1 min read

iOS 12: The MacStories Review ※

Essential reading that will bite off a fair chunk of your calendar. Parents should probably start here instead, though.

Rui Carmo
September 17^th 2018, late night · 3 min read

Fifteen Years of Tao of Mac

Any way I look at it, this site is effectively fifteen years old this year–although it started out under another domain name, it’s been publicly called taoofmac.com since September 18th, 2003, and spotted by Netcraft in November of the same year. For most of that time I’ve written about the Mac, Apple or mobile tech in one form or another, and it’s been a rollercoaster in more ways than one.

Rui Carmo
September 17^th 2018, at night · 2 min read

First Impressions of iOS 12 and watchOS 5

I’m going to be very concise and straight to the point, partly because I’m trying to get some stuff done tonight, and partly because I’m somewhat annoyed.

Rui Carmo
September 16^th 2018, in the morning · 1 min read

Android 9 Pie, thoroughly reviewed ※

I’ve said this a couple of times before, but mobile UX is definitely converging. Were it not for the ragtag Android app ecosystem and Google’s foibles and chaotic relationship with hardware manufacturers, and I could see myself switching to Android full-time (I carry around a company-issued Galaxy 8 these days and hate its default UX with the fire of a thousand suns, and having a vanilla Google experience would be beyond amazing).

Rui Carmo
September 15^th 2018, in the morning · 1 min read

Microsoft acquires Lobe ※

I was a bit taken aback by this. Even though it’s great to see that Lobe was viable, I would have liked to see them succeed on their own, since slotting them into the current Microsoft AI lineup will likely result into their UI becoming just a feature of a larger product and not a standalone, end-to-end experience.

Rui Carmo
September 14^th 2018, at breakfast · 3 min read

On The iPhone Event

I ended up not having the time or inclination to share my thoughts on Wednesday’s iPhone event due to a rather time-consuming (and somewhat depressing) engagement, but now that I’ve regained some zest and vim here is a quick summary.

Rui Carmo
September 13^th 2018, at dawn · 1 min read

Today, Europe Lost The Internet. Now, We Fight Back. ※

The level of incompetence shown by our MEPs and other politicians has now officially crossed over from borderline to full-on, unmitigated evil. I cannot even begin to understand why anyone who lives in this century, was supposed to study the matters at hand and vote on behalf of their constituents’ best interests would throw us into this kind of bottomless pit.

This goes beyond the usual “never ascribe to malice what can best be explained by incompetence” excuse. It is a clear sign that politics as we know it today is still in the Dark Ages. Those people have to be replaced, and the sooner the better.

Rui Carmo
September 9^th 2018, at lunchtime · 2 min read

My Mid-2018 Book Report

I just finished reading Creative Selection, which was a serendipitously great way to cap off this year’s GoodReads challenge (I’ve been reading over 50 books a year for the past 4 years), and it seems like a good point in time to do another book post. Creative Selection is a great book if you’re into product development of any kind, and I wholeheartedly recommend it, but there are many more interesting books out there, so here’s a short report of what I’ve been reading.

Rui Carmo
September 8^th 2018, at breakfast · 1 min read

The 2018 Macbook Pro ※

This makes for an interesting read given the thermal envelope the i9 has and the tests Ole performed. His take on the TouchBar mis-registering touches is a complaint few have articulated so consistently, too.

Rui Carmo
September 4^th 2018, at breakfast · 1 min read

Apple Events and the State of macOS Automation ※

More on Apple Events and security, courtesy of a former colleague of mine. The more I read about this the more it seems that Mojave is going to be security popup land for those of us who rely on UI automation.

Rui Carmo
September 1^st 2018, at breakfast · 1 min read

AppleEvent sandboxing in macOS Mojave ※

Given the amount of scripting I put together the past couple of weeks to do photo processing and what I’m currently fiddling with, this was both serendipitous and highly useful–but I suspect Mojave is going to be a pain to deal with for some people, especially those who rely on both inter-app automation and custom plugins.

It’s not going to be the usual “fallback, update and regroup” update where it regards scripting and automation—there will be some serious breakage due to new security measures.

Apple has always toed a fine line where it regards automation (with little public info towards its evolution and a lot of pessimistic kremlinology doing the rounds whenever features were cut or key people left the company), but macOS is still the best OS in terms of built-in, user-level desktop automation, and I think they know that.

Let’s see how it translates into an iOS-dominated future.

Rui Carmo
August 30^th 2018, late night · 1 min read

PyObjC Using Homebrew On High Sierra (and Mojave)

A few hours ago I was looking for a simple way to build a macOS GUI app and bemoaning the status quo of native app development (as well as the onset of panic due to the end of Summer vacation, but never mind that now…). Having used PyObjC for a good while, it especially irked me that none of my Python runtimes seemed to support it. And now I’ve come across an embarrassment of riches of sorts…

Rui Carmo
August 30^th 2018, at dinnertime · 1 min read

Lenovo’s new Yoga Book replaces the keyboard with an E-Ink screen ※

I like it. Even though the keyboard takes my subconscious deep into TouchBar cold sweat territory, I have to remember I am typing this on an iPad without any qualms.

And would love to get my hands on one, really. The E-ink side would be terrific while traveling (one of my personal gripes is that the iPad mini is too small to read A4 PDFs comfortably and go through all the academic papers I have queued up), but what I find more interesting is that Lenovo feels more innovative than Apple hardware-wise (even though everyone’s fawning over the new Watch leaks as I type this, I can’t help but notice that they have USB-C ports on both sides of this device…).

Rui Carmo
August 30^th 2018, at late afternoon · 2 min read

Racing Against The Clock

I usually take advantage of Summer break to reacquaint myself with real life in various guises, seeing as I’ve always considered it to mark the yearly cycle–a perception that I suppose was ingrained by academia and that I find no reason to do away with seeing as it is still (theoretically, at least) the longest period of time I spend away from “regular” work.

Rui Carmo
August 28^th 2018, at breakfast · 1 min read

SmallCubed MailTags to use Third-Party Sync Server in Mojave ※

Even though I don’t use MailTags, I do rely extensively on Mail Act-On to manage my personal mail, and to be honest I don’t like the idea of merging the entire suite into a single plugin and have it sync data with a third-party server outside iCloud.

I fully get that maintaining tag data inside IMAP is damn near impossible, but adding an extra point of failure and new attack vectors (however unlikely) into my e-mail client is not something I’m interested in at all. I already have too much software that talks to third party machines for no good technical reason whatsoever.

Also, it’s a obvious slippery slope for new features–time to start looking into Mail Act-On alternatives like MsgFiler.

Rui Carmo
August 27^th 2018, at late afternoon · 3 min read

Emergency Photo Triage

One of the yearly chores I keep putting off is triage of Summer photos, and this year is no exception (rather, it is an especially telling occasion, since I have nearly two years of photos to consolidate and file)

Rui Carmo
August 23^rd 2018, in the morning · 1 min read

ROM sites are falling, but a legal loophole could save game emulation ※

As a fan of retro gaming who’s never managed to marshal the time and effort required to build his own home arcade (though not for lack of peer pressure), the most fascinating aspect of the entire rigmarole is not the whack-a-mole routine, but rather the gaping market void that could be filled by the copyright owners themselves.

Nintendo, in particular, has been notable by making a complete hash of its Virtual Console offerings (which I actually bought some of for the Wii), and no console manufacturer has yet taken the time to provide a classic game library that can pass muster. Sure, it would take revenue away from AAA titles–but it would massively increase customer loyalty (as a dad, I would very much like to see my kids playing some classic games) and provide additional revenue streams for publishers.

(Also, I suspect the required engineering effort to port/emulate older games would be much cheaper than what they’re spending on legal fees right now.)

Rui Carmo
August 21^st 2018, at breakfast · 1 min read

This is not your father's Microsoft ※

This is noteworthy because it manages to capture some of the complexity of the multifaceted culture change the company is going through. Being submerged in several aspects of internal culture (which is by no means even or uniform), I can attest to it being a work in progress.

The changes are not evenly distributed yet (and the subsidiary model dampens propagation of change, with various regional and local eddies), but they’re happening.

Rui Carmo
August 20^th 2018, in the afternoon · 1 min read

Computer Vision in Three Lines of Code plus a bunch more lines ※

I love this kind of simple, straightforward solution, especially when it’s written out in a comprehensive fashion.

(I haven’t posted anything about my own machine learning endeavors because they reek of work and are generally fairly uninteresting problems with much simpler business solutions than dialing tech up to 11, but I intend to link to juicy nuggets like this one from now on–I’ve been doing that mostly on LinkedIn, but it’s an audience that values flashiness above substance…)