That’s really interesting! Maybe an experiment with the viterbi algorithm with the goal to maximize similarity with the context (or another score) could also improve results 🤔

Actually using Rust with the Candle library for inference and production code for LLM. Still using python for training, as the majority of the ecosystem is in python. We choose Rust for the intrinsic benefits of the language, and also because we build desktop apps with Tauri. We also use rust for data preparation and other machine learning stuff other than LLMs. If you’re just starting up with Rust, I would recommend gaining more experience before using it as your main language for ML.