From Clouds To Raindrops

The other day I was mulling a particularly boring chore I had just finished (one of the many mundane tasks associated with my industry consulting role) and started musing about what I would really like to be doing with my time.

Don’t get me wrong, I love the management (business, people and processes) stuff–within reason–but I soon started thinking only about engineering stuff, which is what I have been missing the most.

It took no time at all to come up with a bullet list of the points I expand upon below, together with the realization that a fair amount of it is (unsurprisingly) in direct opposition to what I am doing.

But it was still a fun thing to consider, since the kind of stuff I really enjoy doing is actually on two opposite ends of the computing universe:

This is the stuff I did on a daily basis a couple of years ago and miss diving deeper into, because even though the technology involved is still a part of my role, I’m now managing how it’s done for customers, and that is definitely not as much fun as owning the outcomes:

• Building (and automating) massive compute-intensive deployments, typically high-throughput, highly available data munging solutions. This can be anything from off-the shelf Kafka, plain Spark, etc., to custom ML or media conversion pipelines.
• I enjoy the networking and pure compute/infrastructure stuff, which can be engrossing enough when you get down to the details of what each workload needs–and how fast it needs it, not to mention where it needs it. High-throughput, low latency stuff is my jam.
• At a higher level, I love building the data munging logic to build clean datasets–especially in Spark, but I’m no stranger to doing it by hand, especially since there’s often a sweet spot between concurrency approach, compute resources and data representation that just makes it fun.
• As an aside, I’m still trying to reason out why it’s so hard to do “proper” knowledge management with modern techniques. Search and indexing (which I’ve worked in a lot) is just glorified bit stuffing, topic extraction is a bit limited and Latent Semantic Mapping has been commoditised for years but seems to have fallen out of favor, etc.

As to how to build these things today, and the current industry hype–I see containers as mere scaffolding, and Kubernetes as something you need to be absolutely sure you need. I’m becoming hesitant to recommend it wholeheartedly as it’s “solved” many deployment and orchestration problems¹, but, worryingly enough, created a bunch of ceremony and overhead that is on par with peak OpenStack².

Still, for enterprise application workloads, my money is typically on OpenShift, if only because I like to think about it as “Kubernetes for grown-ups” and it’s been quietly making inroads on-premises.

But I digress. There are other things I love dealing with, which lie eat the other end of the spectrum:

I feel a constant need to build tangible stuff, so anything that involves “small computing” or what passes for IoT these days takes up a lot of the time I spend reading about and tinkering with:

• I really enjoy dealing with tiny systems–MCUs, development boards, anything you’d ordinarily classify as IoT or embedded hardware.
• That enjoyment comes partly from the fact that they are constrained systems, and fine-tuning things to run smoothly on tiny machines can be much more enjoyable a challenge than throwing more cores at a problem (and is a skill that scales up).
• There is an entire universe of application deployment and connectivity models to explore there, which also means more interesting problems to solve.
• On the flip side, edge hardware can also be embarrassingly powerful–even an ESP32 can do OCR and face detection these days.

The common theme here, I think, is attention to detail. Yes, you need to marshal the time, people and processes to address each end of the scale properly, and each of those is an extra dimension in its own right.

But since this is solely about technology, I think the best summary (and common ground) regarding what I like doing is hashing out how anything can be done better and simpler–a key question that is unfortunately seldom asked in many places.

I still think about this Calvin and Hobbes strip, though: