2019 in Review

Christmas was nice and relatively quiet (even if there was a lot of packing, furniture assembly and assorted cleaning up involved), so that was a high mark of sorts after a few months of assorted trials.

I’ve been doing a bit of writing on various topics (some of which will find their way here), and I very nearly skipped doing a yearly wrap-up, but in the end decided it was worth taking a stab at it and following roughly the same outline:

Overall, things are mostly the same as last year–cloud adoption is growing, but delivering actual business benefits still takes a long time, due to lack of skills, manpower, agility or (in most cases) plain and simple focus.

Personally It’s still a grind (dozens of day or week-long engagements all over the place), but projects are now a bit bigger and longer-term, which is somewhat gratifying.

I still miss working full-time on international projects and owning the actual end result, but am taking steps to address both and mostly biding my time, although the wait is getting to be somewhat of a grind–and the holiday season adds to the overall sensation of being in a sort of limbo.

In the meantime, things took a couple of dark turns I wasn’t expecting.

I won’t write about the details, but I now have proof positive that ageism and discrimination are very real things in this industry, since I experienced both firsthand and with unusual callousness.

The whole thing was a lot more depressing than I initially realised, but not for long. Rather than dwell on the specifics, I took the time to think about the bigger picture and blame both the “always be closing” mindset and the “warm bodies” mentality of the startup ecosystem, which prioritise enthusiasm and “fresh thinking” rather than guarded optimism and actual experience–two trains of thought that percolated through the market and were latched on by decision-makers who lack the ability to think for themselves (or who suffer from NIH syndrome, I’m not quite sure yet).

But I mostly blame the Portuguese tech scene, which seems to be dominated by near-shoring and services businesses that cater (or report) to other businesses abroad, and few (interesting, but somewhat hype-driven) startups.

Those who know me best can appreciate the fact that, overall, I have become (at least outwardly, and in the broad scheme of things) a lot more restrained and patient than I used to be, largely because there are many other things in life worth worrying about, but looking at actual HR data from a couple of consultancies it’s pretty obvious that ratios and pay grades for Portugal are skewed towards low-end, junior roles.

And knowing the market (and, more importantly, the companies that operate here), it’s a fair assumption to say that we’ve strayed away from long-term value creation (which is what engineering is all about) and are stuck in a quagmire of “consulting” (i.e., body shopping as a service) services and sales.

The net effect is that following my passion for engineering is nigh on impossible at the moment, and I’ve stepped back even further from the lingering notion of setting up my own business.

I spent the year doing a lot of data-related work (mostly Spark and data lakes, which was fun and complex enough to hold my attention), but Kafka has been more prominent of late as customers have suitable foundations to build upon, which is a lot more interesting.

And, of course, there was Building The Future, my trip to Seattle and Pixels Camp, which were certainly the highlight of the year for me.

But I am tired of the data side of things in general, and, as usual, look towards application architecture, infrastructure and hardware for fun.

Kubernetes has taken over (most of my customers, even the more staunchly traditional ones, are using it for new solutions) and its incidental complexity is ballooning to the point where it is getting pushback, which is good.

We’re also still in the explosive phase of its ecosystem expansion, which is a bit surprising–I would have expected the hype to have plateaued by now (it quite likely will by next year), and the cycle to start closing as industry adoption re-shapes it into just another tool.

However, as in every other technology adoption cycle, tools are less than half the story, and I’m concerned that most people I talk to have yet to make the transition to a full-on SRE mindset (and by that I don’t mean reading Google’s books).

I suspect this is a partial reflection of the “cheap resources” mentality that overshadowed the year for me, but there are deeper reasons.

Some people are tied down by corporate structure, others lack the impetus (or the vision) to gradually blend their development and operations teams, but most are just caught up in the more granular portions of the transition—hiring, training, re-architecting their systems, or just getting their CI pipelines to work (a trivial notion for most startups, but in enterprise contexts velocity is more likely to be measured in weeks or even months rather than daily deployments).

I watch these changes unfold bemusedly, providing whatever advice (and fixing) I can, and generally having fun with infrastructure automation of all sorts, even if I have to put in some personal time.

In particular, k3s has been excellent entertainment (even if I have set up and torn down more clusters than actual working solutions with it) and I devote a fair amount of my leisure time to keeping track of who is doing what in that space.

Like I pointed out earlier, most of the work I did this year was focused on data pipelines (plus assorted Hadoop to Databricks migration stuff), and both of these things lie at the inner or outer edges of those pipelines, so a lot was left by the wayside.

IoT, in particular, became more of a solo pursuit since traction for it has decreased locally (at least from my standpoint). I see a lot of ecosystem fragmentation, and (with my telco mindset) wonder when consolidation will come to pass, but were it not for my personal interest in hardware in general and the IoT edge segment that k3s is targeting, I would probably have left the field altogether.

As to machine learning, it has become less of a novelty and (thankfully) more of a process as the shine wears off and people become more sensitive to the need to ensure models are both accurate and refreshed in an orderly fashion (much like I wrote two years ago).

Most of my work has therefore turned to the bigger picture (governance, processes, etc.) rather than actual model creation, but I’m still having some fun with NLP on my own.

Working from home has become a routine affair, so I have been investing some time in making it more effective (and a post is forthcoming on that). But the bottom line is that it has been a major win for me: I find an enclosed space without distractions and constant interruptions to be vastly more productive than open space offices, and my focus time has skyrocketed in terms of quality.

My iMac has become a bottleneck sooner than expected, largely because of its Fusion drive (which, with the benefit of hindsight, was not a good choice). But it has become a hindrance in many ways–Mojave has been a lousy experience in general (with weird, unjustifiable crashes and hangups every now and then), and Apple’s plummeting software quality means that I will likely only upgrade to Catalina by next Spring.

Since I had the foresight of buying an upgradable model, I bumped its measly 8GB of RAM to 24GB over the holidays, which helped a bit.

The highlight of the year, of course, is my iPad mini 5, which has been a great experience throughout (although the Series 5 is a worthy runner-up, and literally quite dear to my heart).

It was an unusual year all around, but I’m really happy about how piku is turning out. I missed running an active Open Source project, although I haven’t managed to devote hardly enough time to it and its little community. That said, and besides my continued forays into home automation, there were a few other things of note.

I keep moving the site around between cloud providers, both to keep skills sharp and to remove any bias I might have towards Azure.

My own micro-PaaS affords me the ability to deploy the app itself on any kind of VM, and I haven’t felt the need to over-complicate matters by moving it permanently to Kubernetes (although I did wrap the engine into a test container and ran it there to my satisfaction, ingress controllers and addressing are still too much of a pain).

Publishing things is now done via git, and thanks to Anders Borum’s amazing Working Copy I can actually post link blog entries via Siri Shortcuts, which is good enough for most of my “impulse” writing.

This year I ran the site for six months in AWS and the rest in Google Cloud, so next year I’m going to move it to smaller, interesting niche providers like Scaleway (since I’ve been meaning to test out their ARM instances). I’ll likely move it back to Azure and start the cycle again later in the year.

I have as of yet nothing substantial to share regarding my music hobby–in fact, I haven’t found the inspiration to practice over the past couple of months, except for a couple of quick hacks for podcast intros that I have yet to finish.

Photography also fell largely by the wayside, too. But I managed to read my usual amount of books this year, which was enough to keep me sane and borderline optimistic.

Last year I decided I was going to try to sleep better–and I did, partly due to this book. Over time, I found a good balance between quality sleep, focus time and not unduly worrying about work, which unfortunately broke down around October (but it was great while it lasted).

Next year I want to do three things:

1. Find that balance again (mind my own time, essentially, focusing on winding down after work)
2. Do interesting stuff (find work I can dream about, not obsess over)
3. Get back into shape (I am currently minding my weight and still walk everywhere, but need to step things up a notch)

I’m perfectly aware that the latter reads like a stereotypical New Year’s resolution, but in my case it’s something I’ve been building up to as priorities shift from having a career to having an actual life.

Life’s too short.