Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)
magazine.sebastianraschka.com
Things I Learned From Hundreds of Experiments
  • twanvl@alien.topBEnglish
    1·
    1 year ago

    Indeed, the choosing alpha as two times as large as r resulted in the best outcomes.

    If I interpret your results correctly, then r=256, alpha=64 actually gives much better results than alpha=512. So here it seems like a slightly smaller alpha is actually better. Or am I missing something?

    • seraschka@alien.topOPBEnglish
      1·
      1 year ago

      =256, alpha=64

      I think I need glasses 😅. You are right that other ratios worked quite well, too, indeed. I amended this section a bit.