You must log in or register to comment.
Try at FP16 or 8bit at most, probably a 13B models suffers too much at 4bits.
I was not impressed with Orca2 13B GPTQ for java and javascript coding type questions. It almost seemed reluctant to give answers. I had to explicitly prompt to print out the code and even then it did not do a very good job.