How to build a diffusion language model

Join the discussion

• Hacker News
• good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.
  by electroglyph
• This should be much more efficient in theory, right? Why dont we see more leading labs adopt this?
  by pwmglenn
• Weak "chain-of-thought" abilities, high error rates and a very bad ability to recover from errors. If they produce a non-sequitur somewhere (which they are highly prone to doing), that global refinement spreads it everywhere like a blood infection superhighway.

  Diffusion text models are cool, but they're functionally much less reliable than autoregressive transformers... and man that's really saying something. Right now most research on them is trying to figure out what complementary systems they need to be reasonably useful.

  by ux266478
• Google was at least trying. Wouldn't be surprised if the others were experimenting with it too. The bar is going to be a lot higher now for for any diffusion model to go from experiment to product since it needs to compete with stuff like glm 5.3 flash and Luna on cost/efficiency for a given quality of output, which is not going to be easy. If it was easy Gemini diffusion would have landed - if it requires a bunch of money and effort it has a much higher bar to make it to market, if it requires some clever breakthrough you have no way of knowing where that's going to come from or what it'll look like/if it'll even seem important when it happens
  by vikramkr
• What's missing from these discussions is the real weakness of diffusion models: you can have two positions where two tokens need to be coordinated, but both spots are teetering between two valid states. They don't always collapse to the matching pair.

  I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.

  by mmastrac
• have u seen the dspark paper, they add a morkov head for light weight dependency, not sure whether it can be extended to multi step refining
  by matrix2596
• I’d like to see more of these models.

  I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.

  In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.

  Even with those caveats, it is very uses-able as a local model.

  by gdiamos
• Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text.

  As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert it to text.

  If you used a single, monospace font, I bet this would be even pretty efficient, because the OCR problem becomes basically just direct template matching.

  But I guess probably there is already a paper out there, I haven't searched. I'd be curious to know if it compares on par with token-based methods.

  by radarsat1
• Diffusion is already being used in drafters
  by faangguyindia
• I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.
  by amelius
• A good video from welch labs on image generation with diffusion models:

  https://www.youtube.com/watch?v=iv-5mZ_9CPY&pp=ygUVZGlmZnVza...

  by rottc0dd
• I've been studying these a bunch for a project in university. Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.

  Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple. I wish some of the blogs/papers I'd read had named "Importance Sampling".

  The probability notation can be pretty confusing too. Sometimes it's hard to understand the "types" of some variables. But I'm inexperienced.

  ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model (which parts aren't completely intuitive), it can do an amazing job filling in the gaps.

  by quirino