Birbla

(Why) Was the LLM breakthrough useful for images, audio, etc.?

(Why) Was the LLM breakthrough useful for images, audio, etc.?

3 pointsby rogerrogerr2 comments

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
    by verdverm
  • It's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929
    by laruss5
  • VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.

    - https://huggingface.co/blog/vlms

    - https://en.wikipedia.org/wiki/Multimodal_learning

    by jerlendds