Rui Carmo
Mercury’s claim of being 10x faster than traditional LLMs is intriguing, especially running over 1000 tokens per second on NVIDIA H100s, but it looks like the kind of paradigm shift I pointed out was needed to make inference more efficient. Couple this with dedicated hardware and I think we might have a winner here.
As someone who’s watched the current crop of AI evolve from the ground up, a shift from autoregressive to diffusion models seems promising, albeit not without its own set of challenges.
The parallel processing that diffusion models afford could indeed streamline reasoning and reduce latency, but applying them effectively to text and code is easier said than done.
If Mercury and Inception Labs have managed to crack that nut, it’s worth a deeper look. The potential to cut inference costs while enhancing output quality could make high-end AI more accessible. However, I’d be cautious until we see how these models perform in diverse, real-world scenarios–and I hope someone is working on an Open Source implementation that can be run locally.