Break the Sequential Dependency of LLM Inference Using Lookahead Decoding | LMSYS Org
lmsys.org

TL;DR: We introduce lookahead decoding, a new, exact, and parallel decoding algorithm to accelerate LLM inference. Look...

  • _Lee_B_@alien.topBEnglish
    1·
    10 months ago

    Hmm, it looks like such a standard linear algebra optimisation that I’m surprised GPUs don’t do it automatically. But yep, looks good, either way.