The Tao of Mac
June 20th 2019, at night · 7 min read
Scheme-ing
Last Sunday I spent a few hours revisiting LISP-related languages, partly because I miss writing Clojure and partly because I wanted to do a relatively simple thing: issue a bunch of HTTPS requests, collate the resulting JSON data and then issue a final POST request. And I wanted to do it with an HTTP library that didn’t suck, in the smallest possible amount of space, and with a static binary. Two out of three wouldn’t be bad, right?
Well, nobody expects getting three out of three in real life, for sure, but as it turns out getting even two out of three is a sizable challenge in modern programming languages.
This particular rabbit hole came about partly because of my k3s and Azure API shenanigans and partly because I have a very similar problem I’ve been tackling with Python and requests, but which needs to run in a more restricted environment.
And it’s always surprising to consider that many people have no idea of what it takes to do an HTTPS request in this day and age–you’d expect that “fetch this URL and parse JSON output” would be the modern day equivalent of “Hello World” in any current programming language, and yet it is often a remarkably tortuous task…
The Usual Suspects
Most of my friends told me “why not just use NodeJS or Go?”.
Well, ignoring the static binary requirement for a moment, because NodeJS is still a complete mess. Just Google for nodejs http request and watch how many times this particular wheel has been reinvented (or better still, look at the change history for the built-in http library). If I had to do it in NodeJS, I’d use a wrapper like request for the sake of sanity and readability, at the expense of instantly sucking (pun intended) in a bunch of dependencies that really ought to be built-ins…
Don’t believe me? Look at the core https version, which requires an explicit data pump:
const https = require('https');
https.get('https://endpoint', (resp) => {
let data = '';
resp.on('data', (chunk) => { data += chunk; });
resp.on('end', () => {
console.log(JSON.parse(data));
});
}).on("error", (err) => {
console.error(err.message);
});
…and the request version, which is vastly more readable:
const request = require('request');
request('https://endpoint', function (error, response, body) {
if(error) {
console.error('error:', error);
} else {
console.log(JSON.parse(body));
}
});
…and I’m not even going to get into customizing request headers and cookie handling (which is where request really makes things easier for me), or how works the same way for buffers, streams, etc.
As to Go, for all its merits, simple things like this are comparatively harder and error-prone to write.
For instance, here’s part of what I’ve been working on (still largely untested, mind you) to port those 7-line Python requests I did against the Azure instance metadata APIs. And note how much more cumbersome the whole thing is when (for extra brownie points) you start parsing a nested JSON document using struct annotations for unmarshaling:
import (
"encoding/json"
"io/ioutil"
"log"
"net/http"
)
type InstanceMetadata struct {
Compute struct {
ResourceGroupName string `json:"resourceGroupName"`
SubscriptionID string `json:"subscriptionId"`
} `json:"compute"`
}
func getInstanceMetadata() *InstanceMetadata {
client := &http.Client{}
req, err := http.NewRequest("GET", "http://169.254.169.254/metadata/instance", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Add("Metadata", "true")
q := req.URL.Query()
q.Add("api-version", "2018-10-01")
req.URL.RawQuery = q.Encode()
resp, err := client.Do(req)
if err != nil {
log.Fatal(err);
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err);
}
metadata := InstanceMetadata{}
err = json.Unmarshal(body, &metadata)
if err != nil {
log.Fatal(err)
}
return &metadata
}
And it’s about as crufty in Rust, which I find interesting but hard to justify against the sprawling Go ecosystem.
Looking for Options
So no, this isn’t the kind of developer experience I want for my own projects, even if it is one that has pretty much taken over the entirety of modern “systems” programming over the past few years.
For my hobby stuff, I’m looking for a more concise language (hence more likely to be dynamic/interpreted) that I can read and write without so much hassle, and that brings some fun into the process.
And I miss LISP (I am one of those people who actually enjoy it because it is an intellectual antidote for drudgery, even if I do love Python and Go).
So I decided to use that particular problem I had to solve as an excuse to check out the state of the art in that corner of the computing universe. I’ve been tracking a lot of Scheme implementations over the years including a few that compile down to C or native code, so it was time to see how useful they were.
All the tests below were run on Ubuntu 18.04.2, both in WSL and a standalone machine I used for testing standalone binaries (I also used a 16.04 machine on occasion), since it’s usually the target Linux I build for.
Fennel
Version: 0.2.1, installed via
luarocksatop Ubuntu’s standard Lua 5.1
Fennel is not the first thing I tried, but it was the first one that nearly hit the spot. It is a LISP that compiles to Lua, and it does so in a quite elegant way, turning this:
(local http (require "ssl.https"))
(local ltn12 (require "ltn12"))
(local inspect (require "inspect"))
(fn request [url]
(local resp {})
(let [(res code headers)
(http.request
{ "url" url
"method" "GET"
"sink" (ltn12.sink.table resp)
"headers" {"User-Agent" "lua"} })]
(print (inspect resp))))
(request "https://endpoint")
…into this:
local http = require("ssl.https")
local ltn12 = require("ltn12")
local inspect = require("inspect")
local function request(url)
local resp = {}
do
local res, code, headers = http.request({headers = {["User-Agent"] = "lua"}, method = "GET", sink = ltn12.sink.table(resp), url = url})
return print(inspect(resp))
end
end
return request("https://endpoint")
The results are pretty decent considering that ltn12 is the kind of thing you can seldom avoid in Lua and that the resulting code is pretty much idiomatic if you ignore the lack of human-crafted whitespace.
The above does not do any JSON parsing, but there is also a requests rock for Lua, which brings in a few more dependencies (xml and cjson) and makes everything even more readable:
(local requests (require "requests"))
(local inspect (require "inspect"))
(let [res (requests.get "https://endpoint")]
(print (inspect (res.json))))
…becomes:
local requests = require("requests")
local inspect = require("inspect")
do
local res = requests.get("https://endpoint")
return print(inspect(res.json()))
end
Fennel is a thing of concise beauty, and I can (theoretically) compile everything down to a ~1MB executable (which is the rough footprint of the Lua runtime plus dependencies) with tools like luapak.
But I could not manage to get luasec to play along, otherwise I would have likely stopped here.
It is very tempting, though, since using Lua would also make it possible for this to run on an ESP8266/ESP32, which would be perfect for another use I have in mind.
Chicken Scheme
Versions: 4.12 (shipping with Ubuntu), 5.1 (built from source)
My next stop was Chicken Scheme, an old acquaintance that compiles down to C and does everything I need rather elegantly–the simplest possible HTTPS and JSON decode call looks like this on Chicken 4:
(use http-client json)
(pp (with-input-from-request
"https://endpoint"
#f
json-read))
The above compiles down to a dynamic executable that is only 35112 bytes and obviously depends upon a number of dynamically-linked libraries. But not just the ones it reports via ldd:
% ldd request
linux-vdso.so.1 (0x00007fffc34bf000)
libchicken.so.8 => /usr/lib/libchicken.so.8 (0x00007f0f8d830000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f8d430000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0f8d090000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0f8ce80000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0f8e600000)
What stood out to me was that this does not include openssl nor the extensions I had to install in order to get it to build:
export CHICKEN_REPOSITORY?=$(HOME)/.chicken
build:
csc request.scm
deps:
chicken-install \
openssl \
http-client \
json
init-repo:
chicken-install -i $(CHICKEN_REPOSITORY)
Running the executable with strace and filtering the output to capture all the openat calls reveals it dynamically loads around 17MB of libraries, so that would be its maximum footprint.
As to building a re-distributable binary, trying to build with -static and the openssl extension as a dependency didn’t work out in Chicken 4 (which is what I had handy), but I got a bit closer today with Chicken 5–I managed to get some libraries linked in, but not all of the extensions I was using.
Racket
Versions: 6.11 (bundled with Ubuntu), 7.3 (via this ppa)
As far as Schemes go, Racket is a very popular choice (and Ubuntu 18.04 ships with 6.x packages), so I had a quick stab at it:
#lang racket/base
(require net/url)
(require json)
(define (get-json url)
(call/input-url (string->url url) get-pure-port read-json))
(print (get-json "https://endpoint"))
And, surprisingly, it was the only one that gave me a set of re-distributable files with nearly zero hassle.
The above builds down (using raco exe) to a 6MB file that still requires racket to run, and that can be packed (using raco distribute) into a roughly 10MB bundle:
dist
├── [4.0K] bin
│ └── [6.0M] request
└── [4.0K] lib
└── [4.0K] plt
├── [3.9M] racket3m-6.11
└── [4.0K] request
├── [4.0K] collects
└── [4.0K] exts
└── [4.0K] ert
└── [4.0K] r0
└── [1016] dh4096.pem
More to the point, it just worked when I copied the files to a “blank” machine and ran the executable, and was the only solution so far that did so (in fact, both versions did, with marginal differences in final binary sizes).
Chez
Version: 9.5 (shipping with Ubuntu)
Next up was Chez Scheme, which is becoming the underpinnings of Racket 7.x releases, but where I drew an almost complete blank–because Chez, despite shipping with Ubuntu as well, currently lacks a comprehensive set of libraries (although thunderchez and scheme-lib have plenty of material to go through…).
Gerbil & Gambit
Versions: 4.9.3 (Gambit), 0.15.1 (Gerbil)
A few days later, I decided to take a look at Gerbil, which is a nice wrapper around Gambit Scheme that happens to oriented towards systems programming.
Gerbil wraps Gambit in much the same way Racket wraps Chez, but with a more pragmatic, low-level twist in its libraries.
As a result, my little sample program was quite straightforward to get going:
(import :std/net/request)
(import :std/text/json)
(export #t)
(def (main)
(displayln
(json-object->string
(request-json
(http-get "https://endpoint")))))
As to my requirement for binaries, building one with an explicit linkage to libssl and other system libraries was relatively easy:
% ldd request
linux-vdso.so.1 (0x00007ffffbef7000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f10ecac0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f10ec8b0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f10ec510000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f10ec280000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f10ebdb0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f10eb9b0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f10ed600000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f10eb780000)
Building a fully static binary, however, wasn’t something I managed to do inside of a couple of hours, since it requires getting to grips with both Gambit and the wrappers Gerbil puts in place and I kept coming up against lack of documentation in that regard.
Conclusion
I’m going to be looking at Fennel and Chicken again in the near future, the former because it affords me the ability to target other architectures and the latter because I suspect I will be able to build a fully standalone (and insanely fast) executable with Chicken 5.x, and I like its portability.
In the meantime, nothing seems to beat Racket for “just working” on Intel architectures, and I will be setting up 7.3 for my little project, and I recomend it if you want to get started quickly and want something that is very thoroughly documented.
But I’m going to dive into Gerbil afterwards, since it is likely to be more suitable for my needs in the long run–and I’ve already started building multi-architecture Docker containers for it so I can try it out.
A Minor Note Regarding VS Code
In other news, I recently switched from VSCodeVim to amVim because I found VSCodeVim hung the editor randomly.
But, more relevant to this post, I came across Calva while searching for a barebones paredit mode (which it ships as a separate extension).
I still prefer vim‘s paredit (it’s wired into my typing reflexes by now), but it is good enough to deserve a mention.
