While instructing fine tuning mistral, what are the parameters that needs to be set for the tokenizer?
what is the default eos token for Mistral and padding should left or right?
I’ve exhausted all the online articles trying to find this. Please help. I’m instruction fine tuning Base Mistral for Text to SQL task.
I believe you are using LoRa? How are you training? What library are you using? In my experience (which is limited) many libraries don’t set attention to 1 for the eos token. Thus the model is trained to ignore it. If you use the hugging face trainer library, you need to define your own mapping function in which you set the attention for the eos token to be 1. Make sure your dataset used for training also uses at the end of the response. If you do that, then you probably don’t need to mess with the attention. All these problems go away when you use an instruct model as it’s already trained to stop at the end. If you use the same prompt format in your fine tuning dataset, that will work well.