Testing AI and LLM on Rockchip RK3588 using Mixtile Blade 3 SBC with 32GB RAM ➹

This is great news for my own Rockchip chipset exploration, which still has a ways to go–there now seems to be working Mali GPU acceleration for LLMs, and having more people doing this kind of testing on ARM is both informative and a sign there’s interest in the small model, edge AI scenarios I’ve been toying with.

Serendipitously, I did look at llm-rk3588 when I got my Orange Pi 5+ (it was actually developed on one) but discarded it because the NPU can’t really be used for LLMs and the required firmware blob didn’t load under my Armbian build (I assume the repo owner was using the Orange Pi linux distro).

I would prefer that be baked into ollama to have a baseline that is comparable with Apple Silicon and Intel chips–i.e., exact same model weights, at least–but its great to see something can be made to work with the Mali GPU (although I’m not really clear on what model layers will benefit, how it deals with quantization, etc.).

But I will have a look at reproducing (and, preferably, building locally and tweaking) what was used here and try to give both CPU and GPU inference a go on a similar (but more compact) 32GB RAM board that I have sitting on my in tray.

So watch this space (I’ve been busy with a lot of work and more physical testing)…