What are Diffusion LLMs?
A video showing the speed of a diffusion LLM at generating some demo code versus a standard LLM.
A startup called Inception Labs have applied the same type of AI algorithm that's used to produce images (known as diffusion) to producing text to create something called a Diffusion Large Language Model (dLLM).
Rather than predicting the text a word at a time it produces the whole thing in one go and then iterates on it to improve it in the exact same way that an image generation algorithm starts with a rough blur and gradually fills in more detail until it has a high definition image.
This solves some of the key issues with Large Language Models as they currently exist - by predicting the whole thing in one go the text is able to take account of what comes later in the document. The reason LLMs suck at writing jokes is because they're unable to come up with the punch line before the set up - dLLMs should be able to handle that.
Also, with current LLMs once a word is written there's no way of changing it so if a model makes a mistake it doesn't have the ability to recognise and correct it - currently we have to use prompting techniques such as self critique to try and get around this but dLLMs should automatically pick up on issues and fix them as part of their initial generation of the text.
Finally, in the same way that you can force an image generation algorithm to produce a specific type of image by setting the initial starting image, you should be able to do something similar with dLLMs which should make it easier to enforce specific styles on the text you're generating eg code
They can also be much faster and more efficient - Inception Labs claim that their coding model can produce an answer 10x faster and cheaper than a standard LLM.
But dLLMs are still in their infancy - why should you care?
When Deepseek launched (what feels like years ago but was really just a few weeks) there was a major hullabaloo as people (especially investors) had to readjust their expectations of what it takes to build a successful AI model.
dLLMs are another example of techniques that are reasonably well understood being applied to Large Language Models with significant impact on the efficiency and capability of the models. There are plenty more of these in the pipeline and it's reasonable to expect AI to still get many orders of magnitude cheaper and more powerful over the next couple of years.