Site Engine Update

It’s been since I moved this site to Azure static storage, and I think a few notes are in order.

The site is still powered by my Wiki engine and I still keep it updated by tossing files into a git repository, but it now has even more unique twists.

Current Setup

TL;DR: Instead of rebuilding the entire site whenever a single page changes (or even syncing generated content), my current engine does delta updates directly onto an Azure storage container upon git push.

Or, to put it another way, it:

  • Takes the deltas off that push.
  • Matches them against what is currently published in $web.
  • Figures out what (if any) images, related pages and back-links need updating via a small SQLite database that holds all those references.
  • POSTs the generated HTML for any updated pages directly to a $web public container (which it does wickedly fast thanks to my own asyncio library for talking to Azure Storage).

This contrasts with the approach of many static site generators of rebuilding the entire site, saving the HTML to a folder and then syncing all of that to a public bucket/container, which can be pretty I/O intensive.

Or (like many people do these days with GitHub Actions et al) spawning and provisioning an entire VM to do so.

I don’t like either, and find the “pseudo-CI” approach a tad lazy (and wasteful).

Bang For The Buck

While it’s true that the site generator is on an “always on” (and admittedly pokey) Azure B1ls instance (1 VCPU, 512MB RAM), that machine is shared with a bunch of other small services, and the builder service is actually removed from RAM when idle (thanks to Piku/uwsgi magic), so overall costs are negligible.

The process has minimal storage and I/O impact (it only needs to work with a copy of the raw content and a SQLite database with page relationships), and takes less than 10 seconds to generate and publish a new page (or even a few hundred of them) from the moment I type git push, which compares well with my age-old, -driven “live” syncing approach.

I could run the whole thing on a Pi Zero (and yes, I’ve tested it on a 3A+, just because the Zero is quite slow for full rebuilds), so that’s fine.

Pluses

There are a few distinct advantages with this setup:

  • No more worrying about HTTP handling and the constant flurry of botnets trying to hack their way into something that was never a Wordpress site in a rather heavy-handed way.
  • Much better availability as there would occasionally be some OS weirdness, VM update or configuration tweak that would take the site offline until I noticed.
  • Maybe a smidgeon more performance (even considering I had optimized HTTP request handling and have been using Cloudflare for years, the site does feel a tad snappier).

Minuses

There are a few things I’m not happy about, though:

  • My “magical” Wiki URL redirection feature was replaced by a gnarly bit of that I’m not exactly enamoured with either. I might push that back to a Cloudflare worker, but am wary of relying too much on any single service.
  • I’m not particularly impressed with DuckDuckGo site search. Even after fiddling with various things like Bing Webmaster tools (since DuckDuckGo gets part of its catalog from it), search is just not good enough. I publish a complete sitemap and have OpenGraph everywhere, but their crawler simply doesn’t do a good a service as my old SQLite-powered full text indexer, and it’s getting very annoying.
  • The engine is a bit slow when rendering the full working set of 8000+ pages (when I do layout changes, for instance). can only do so much, and even though I pre-bake a lot of stuff, I’d like it to be a bit faster when parsing and rendering the actual pages.

I’ve been putting off using Azure Search because I don’t want to have to maintain a search endpoint, but I might end up doing it regardless.

Nice To Haves

There are a few minor things I’d like to have done, though:

  • I still haven’t pushed out the code to render archives, which has an annoying bug I haven’t had the time (or willpower) to fix because it will eventually force me to break a couple of longstanding abstractions. Update: I decided to not overthink it and just use my old Wiki code to generate archive pages.
  • OpenGraph can be improved little further, especially where images are concerned. Adding better image descriptions and the like feels like something I might have some fun doing, so will probably be adding Azure Cognitive Services to my rendering pipeline
  • I would love to have the builder run as an Azure Function, but there is simply no way I can get git to run inside one, and Piku has rendered this kind of service so simple and easy to maintain that I don’t feel the urge to tackle the amount of extra complexity required to work around that.

Engine Maintenance

Finally, the current codebase could do with a little cleaning up, since it has suffered a bit of feature creep over the years.

It’s relatively small at ~4000 LOC (discounting templating and HTML), but the actual static generator is a ~1000 LOC bolt-on that could be streamlined and blended in a little better, so I’d like to trim the whole thing back into the ~3000 LOC range.

And, in general, I just wish the entire codebase was simpler, for is a wonderful language but still lacks the conciseness of –and I would very much like to go back to something like the Hy implementation I had a few years back1.

And yet, replacing would be a tall order indeed. There are a few interesting candidates (Janet, Fennel, , and even ), but none of them have the full range of wonderful libraries I currently rely on.

I do, however, hear the siren call of building something leaner and meaner, so I might do some experiments over summer break.

Not Going To Happen

There are a few things I most definitely won’t be doing, though:

  • Building a Docker container (again, Piku has made things so easy to deploy and update that setting up a private registry, a full-blown CI/CD pipeline and whatever else would feel like a chore).
  • Getting this to run on Kubernetes (ha!).
  • Porting the engine to (I gave it a go and realized that implementing the XML transformation pipelines I have would be a bit of a chore).

So, in essence, things are stable, reliable and with enough headroom and small things to tweak to keep being interesting, but definitely low impact.

I might do a redesign, though. It’s about time to see if there is something else I can do layout-wise (although I do like Georgia‘s readability and universal reach, and the lack of any unnecessary frills).


  1. Besides wanting to move to Python 3, another reason I was that Hy kept repeatedly breaking over time and I’d need it to settle into some sort of long-term stable form first. They’re about to go 1.0 (which took its time), but I am definitely going to wait and see in that regard. ↩︎

Rabbit Holes I Dive Into Every Now And Then (Alternative Operating Systems Edition)

Over the past few years, and as part of my perennial personal quest for a nicer computing experience, I keep finding myself going down a small set of technological rabbit holes, which I’m listing here in an attempt to remind future me that I might already done so too many times.

Read More...

Three Thoughts on WWDC 2021

So I finally watched the keynote and the State Of The Union1, and came away with three main thoughts:

Read More...

The Big Sur kernel_task Troubleshooting Saga

Following regarding my kernel_task woes (which began nearly two weeks ago now), I decided to strip out the updates from that post and consolidate them here with a little more detail.

Read More...

The Arturia KeyLab Essential 61

Although I have nothing to show for it yet, my music hobby is getting serious enough that in my office. This because even though we already have a very nice , there is no way I can use it with Logic from several rooms away1.

Read More...

Sucking Up to kernel_task

Just after upgrading to Big Sur 11.4, has taken to stall every now and then with kernel_task taking up most of the CPU, which is tremendously annoying as it becomes unresponsive even when I’m using it as a souped-up thin client to access my corporate virtual desktop environment.

Read More...

Notes on Wireless MIDI

It’s getting warm enough that a laptop (or a fan-happy desktop) is a nuisance do deal with, which together with an increased propensity to fall asleep on a couch means I’m back mostly using my during weekends.

Read More...

Home Automation Catch-Up

As the weather has now improved beyond the need to turn the heaters on, I temporarily decommissioned most of my Sonoff S20 sockets until next winter and thought it was a good opportunity to catch up on my home automation endeavors.

Read More...

Cubasis 3.3 and the KORG nanoKONTROL Studio

The I was futzing around pondering getting a better controller that I could use both with my desk setup and my iPad and, since I am not keen on buying stuff with proprietary protocols (regardless of how nice they look, I prefer stuff that won’t rely on the manufacturer’s own software) learned about Mackie Control and HUI and how they are supported by Arturia products.

And as it turns out Cubasis 3.3 came out recently, and besides extremely comprehensive learn support, it also supports Mackie and HUI.

Since I have a KORG nanoKONTROL Studio, I decided to investigate if it could work that way.

After a few tries, I figured out that if I booted the controller in Pro Tools mode (by turning it on while holding down Scene and Track Left-Arrow), you can enable it in MIDI settings like so:

Pretty trivial to set up, really.

After you’ve done that, everything (jog dial, transport controls, and all the track controls/pan/faders) just works, which probably makes Cubasis the best iOS DAW right now1.

It also gives me firmer notions as to what kind of keyboard/controller I might get (Arturia’s are currently at the top of my list, but the interesting models are several years old and lack both Bluetooth and aftertouch…).


  1. GarageBand has zero MIDI learn features, and NanoStudio 2 doesn’t seem to have either Mackie or HUI support, but I might be wrong as it at least has usable MIDI learn. ↩︎

Spring Cleaning

I keep telling myself that “boring is good”, and yet things have been so far from boring that a moderately calm week like the past one is an outlier. I’m still mostly from work, but some geekery has taken place, albeit not particularly exciting stuff.

Read More...

Stop

After plodding through I decided to take this last Friday off, with the not-quite-amazing outcome of spending most of it sleeping and reading The Economist in an exquisitely sunny living room that I had a vague recollection of having in our house.

Read More...

Pause

Over the past few months, the 10h-a-day pace that became a constant during the pandemic started creeping up as I took up a new role1 and needed more and more time to get work done after a day’s worth of meetings. I ended up working the last couple of weekeends as well, so this one was almost exclusively devoted to downtime.

Read More...

Archives