I’m looking for insights and advice on extending the context window of the LLMs (most specifically Mistral).

Whether you’re a researcher, developer, or enthusiast in the field, I’d love to hear about your experiences and recommendations. Are there any specific techniques, methodologies, or tools you’ve found effective in extending the context window for LLMs?

Additionally, if you’ve encountered challenges in this area, how did you overcome them? Any resources, papers, or community discussions you can point me to would be greatly appreciated.

  • sshh12@alien.topBEnglish
    1·
    2 years ago

    I’ve been working on some experimental context window extensions using multimodal models https://github.com/sshh12/multi_token

    Similar to the idea of putting text into an image for GPT4V, I’m just directly encoding chunks of texts into embeddings and injecting them in the models. This gives you a very lossy 128x extension of your context window which is pretty massive.