Last night after work I decided to see how easy it would be to run a Takahē ActivityPub instance under Piku, my tiny Python-oriented PaaS.
Self-hosting Mastodon is all the rage, but having to deal with a full-blown installation of Ruby (which is always a pain to install properly, even if you use
rbenv), plus the abomination that is Sidekiq and the overall Rube Goldberg-esque architectural approach that is almost mandatory to deal with the complexities of ActivityPub is just something I don’t want to maintain. Ever. Even inside Docker.
Which is why I have been developing my own ActivityPub server using Sanic and a very lightweight
asyncio-based approach at handling all the transactional aspects of ActivityPub atop SQLite. And let me tell you, I honestly wish the protocol was less about doing what boils down to P2P webhooks with PEM signatures embedded in requests.
But Takahē is now aiming to support client apps as of version
0.6, is built on Django (which I have always loved as a framework), and it saves me from the trouble of building everything from scratch, so… I had to try it out.
More to the point, Django is exactly what Piku was originally designed to run.
Besides running as a
WSGI app, Takahē uses an async stator to handle all the background tasks (which is also exactly the pattern I aim for and designed Piku to support), so I just had to see how easy it was to get it running under Piku on very low-end hardware.
I have a 4GB Raspberry Pi 4s set up as an SSD-backed Proxomox server, hosting several different
arm64 LXC containers I use for developing stuff. I love it because I can use LXC CPU allocations to throttle things and make sure they run fast enough on very low-end hardware, plus I can just snapshot, mess up and restore entire environments.
So I set up an Ubuntu 22.04 container with 1GB of RAM and access to 2 CPU cores, capped to 50% overall usage–which is roughly the performance of a Raspberry Pi 2 give or take, albeit with a fully 64-bit CPU.
I deployed Piku, set up a CloudFlare tunnel, and then went to town.
Zero Code Changes Required
In short, what I needed to get Takahē up and running under Piku was to:
- Clone the repository.
- Create a
productionremote pointing to Piku.
- Edit the supplied
- Do a
git push production main.
It was that simple.
Here’s the configuration I used, annotated. First the
# Yes, I went and got it to use SQLite, and it nearly worked 100% TAKAHE_DATABASE_SERVER=sqlite:////home/piku/takahe.db # This is what I eventually migrated to (more below) # TAKAHE_DATABASE_SERVER=postgres://piku:<password>@localhost/takahe # I actually love Django debugging, and with it on I can see the inner workings TAKAHE_DEBUG=true # You know who uses this password, don't you? TAKAHE_SECRET_KEY=pepsicola # No, it's not the one I'm actually using. # Anyway, this next one breaks a little on Piku, so I need to revise parsing for this case. TAKAHE_CSRF_TRUSTED_ORIGINS=["http://127.0.0.1:8000", "https://127.0.0.1:8000"] TAKAHE_USE_PROXY_HEADERS=true TAKAHE_EMAIL_SERVER=console://console TAKAHE_MAIN_DOMAIN=insightful.systems TAKAHE_ENVIRONMENT=development TAKAHE_MEDIA_BACKEND=local:// TAKAHE_MEDIA_ROOT=/home/piku/media TAKAHE_MEDIA_URL=https://insightful.systems/media/ TAKAHE_AUTO_ADMIN_EMAIL=<my e-mail> # I also added a Redis server later, without any relevant impact in RAM usage TAKAHE_CACHES_DEFAULT=redis://127.0.0.1:6379/0 SERVER_NAME=insightful.systems # This is all Piku config from here on down # I need IPv6 off for sanity inside Proxmox DISABLE_IPV6=true LC_ALL=en_US.UTF-8 LANG=$LC_ALL # This ensures nginx only accepts requests from CloudFlare, plus a few extra tweaks NGINX_CLOUDFLARE_ACL=True NGINX_SERVER_NAME=$SERVER_NAME # These are caching settings for my dev branch of Piku NGINX_CACHE_SIZE=2 NGINX_CACHE_TIME=28800 NGINX_CACHE_DAYS=12 # This has nginx cache these prefixes NGINX_CACHE_PREFIXES=/media,/proxy,/static/admin # This maps static user media directly to an nginx route NGINX_STATIC_PATHS=/media:/home/piku/media,/static:static-collected,/robots.txt:static/robots.txt PORT=8000 # You want to set these, trust me. I should make them defaults in Piku. PYTHONIOENCODING=UTF_8:replace PYTHONUNBUFFERED=1 TZ=Europe/Lisbon # This tells uWSGI to shut down idle HTTP workers # Saves RAM, but startup from idle is a bit more expensive CPU-wise UWSGI_IDLE=60 # We need to run at least 2 uWSGI workers for Takahe UWSGI_PROCESSES=2 # Each worker will have this many threads # (even though I'm only giving this 2 cores) # to match the original gunicorn config. UWSGI_THREADS=4
…and only very minor changes to the
wsgi: takahe.wsgi:application worker: python manage.py runstator release: python manage.py collectstatic --noinput; python manage.py migrate
In essence, I removed
gunicorn (which I could use anyway) to let
uWSGI handle HTTP requests and scale down to zero (saving RAM). And yes, Piku also supports
release activities, thanks to Chris McCormick.
And that was it. Zero code changes. None. Nada. And I can use exactly the same setup on any VPS on the planet, thanks to Piku.
After a little faffing about with the media storage settings (which I got wrong the first time around, since Takahē also uses
/static for its own assets), I had a fully working ActivityPub instance, and, well… John Mastodon just happened to sign up:
Takahē nearly works with SQLite, but sadly it relies on
JSON_CONTAINS, which is an unsupported feature in SQLite (but one which PostgreSQL excels at).
The upshot of this was that the stator
worker was very sad and bombed out when trying to handle hashtags–but all critical stuff worked, so there might well be a workaroud.
But I took some time after breakfast to migrate the database, and since my Django skills are rusty, here are my notes:
# Open a shell to Piku ssh -t [email protected] run takahe bash sudo apt install postgresql python manage.py dumpdata > /tmp/dump.json sudo su - postgres psql
-- Set up the database create user piku; create database takahe; alter role piku with password '<mysecret>'; grant all privileges on database takahe to piku; alter database takahe owner to piku;
# Reset all the migrations, just in case find . -path “*/migrations/*.py” -not -name “__init__.py” -delete find . -path “*/migrations/*.pyc” -delete # Reapply them python manage.py makemigrations python manage.py migrate # Wipe all default entities python manage.py shell from django.contrib.contenttypes.models import ContentType ContentType.objects.all().delete() # Load everything back python manage.py loaddata /tmp/dump.json
Overall, I’m quite impressed with the whole thing. Even with such measly resources and Linux’s tendency to take up RAM with buffers, Takahē under Piku is taking up around 100MB per active worker (2 web handlers, plus the stator worker), plus less than 50MB for PostgreSQL and
So I’m seeing less than 512MB of RAM in actual use, and a steady <10% CPU load inside the container as the stator keeps picking up inbound updates, handling them (including any outbound requests) and doing all the messy housekeeping associated with ActivityPub:
But here’s the kicker: Since this is being capped inside LXC, that is actually around 5% overall CPU load on the hardware–which should translate to something like 2% of CPU usage on any kind of “real” hardware.
With only one active user for now (but following a few accounts already), this is very, very promising.
I have no real plans to leave
mastodon.social for my own domain, but using Takahē to host a small group of people (or a company) with nothing more than a tiny VPS seems entirely feasible, and is certainly in my future.
Right now, I’m going to try to contribute by testing various iOS clients (I will be using the Takahē public test instance as well) and do some minor tweaks to my install, namely:
- Setting up
nginxcaching. Cloudflare is already caching one third of the data, but I want to bulk up this setup so that I can eventually move it to Azure, and I’ve been meaning to add that to Piku anyway.
- Fine-tuning the stator to see how it scales up or down (I might want to try to scale it down further).
gunicornto see if it makes any difference in overall RAM and CPU.
- Seeing if I can get it to work on Azure Functions (that is sure to be fun, although the current SDK failed to install on my M1 and I haven’t tried since).
- Look at how media assets are handled and see if I can add a patch to support Azure Storage via my own
- Deploy on my k3s cluster, to get a feel for how much it would cost to run on spot instances.
There goes my holiday break, I guess…
Update: A Few Days Later
I’ve since sorted out
nginx caching in Piku (and will soon be merging it to
main), which makes things significantly snappier. I’ve also filed #287 to improve caching via Cloudflare and #288 to have
nginx immediately cache assets (which works for me, at least).
Before that, I had some fun tuning stator pauses and filed #232, which resulted in a tweak that lowered idle CPU consumption to a pretty amazing 3% in my test instance.
With the caching tweaks,
gunicorn doesn’t have any real advantage against
uWSGI workers, although I suspect that may be different in higher-load instances.
I’ve also tossed the source tree into an Azure Function and got it to “work”, but not fully. Right now I’m not sure that is worth pursuing given I still need an external database, but I’m really curious to try again in a few months’ time.