Rui Carmo
I think this both vindicates my year-long interest in running LLMs on industrial “edge” hardware and signals that it’s time to step back and re-assess how to address that space.
3.2 TOPS isn’t exactly stellar performance when compared to what you can (nominally) get out of a RK35xx’s built-in NPU and way below the target for things like Copilot PCs, but the fact that we’re starting to get what is effectively commodity hardware at very low wattages changes the game a bit–AI is going to get cheaper and cheaper to deploy at the edge, and I don’t expect that trend to be limited to low-end hardware.
The stated compatibility with various small LLMs and the promise of future updates is commendable, but the real test (and hints at solution TCO) will be in short-term execution and software support, as always.