I posted my latest LLM Comparison/Test just yesterday, but here’s another (shorter) comparison/benchmark I did while working on that - testing different formats and quantization levels.

My goal was to find out which format and quant to focus on. So I took the best 70B according to my previous tests, and re-tested that again with various formats and quants. I wanted to find out if they worked the same, better, or worse. And here’s what I discovered:

Model Format Quant Offloaded Layers VRAM Used Primary Score Secondary Score Speed +mmq Speed -mmq
lizpreciatior/lzlv_70B.gguf GGUF Q4_K_M 83/83 39362.61 MB 18/18 4+3+4+6 = 17/18
lizpreciatior/lzlv_70B.gguf GGUF Q5_K_M 70/83 ! 40230.62 MB 18/18 4+3+4+6 = 17/18
TheBloke/lzlv_70B-GGUF GGUF Q2_K 83/83 27840.11 MB 18/18 4+3+4+6 = 17/18 4.20T/s 4.01T/s
TheBloke/lzlv_70B-GGUF GGUF Q3_K_M 83/83 31541.11 MB 18/18 4+3+4+6 = 17/18 4.41T/s 3.96T/s
TheBloke/lzlv_70B-GGUF GGUF Q4_0 83/83 36930.11 MB 18/18 4+3+4+6 = 17/18 4.61T/s 3.94T/s
TheBloke/lzlv_70B-GGUF GGUF Q4_K_M 83/83 39362.61 MB 18/18 4+3+4+6 = 17/18 4.73T/s !! 4.11T/s
TheBloke/lzlv_70B-GGUF GGUF Q5_K_M 70/83 ! 40230.62 MB 18/18 4+3+4+6 = 17/18 1.51T/s 1.46T/s
TheBloke/lzlv_70B-GGUF GGUF Q5_K_M 80/83 46117.50 MB OutOfMemory
TheBloke/lzlv_70B-GGUF GGUF Q5_K_M 83/83 46322.61 MB OutOfMemory
LoneStriker/lzlv_70b_fp16_hf-2.4bpw-h6-exl2 EXL2 2.4bpw 11,11 -> 22 GB BROKEN
LoneStriker/lzlv_70b_fp16_hf-2.6bpw-h6-exl2 EXL2 2.6bpw 12,11 -> 23 GB FAIL
LoneStriker/lzlv_70b_fp16_hf-3.0bpw-h6-exl2 EXL2 3.0bpw 14,13 -> 27 GB 18/18 4+2+2+6 = 14/18
LoneStriker/lzlv_70b_fp16_hf-4.0bpw-h6-exl2 EXL2 4.0bpw 18,17 -> 35 GB 18/18 4+3+2+6 = 15/18
LoneStriker/lzlv_70b_fp16_hf-4.65bpw-h6-exl2 EXL2 4.65bpw 20,20 -> 40 GB 18/18 4+3+2+6 = 15/18
LoneStriker/lzlv_70b_fp16_hf-5.0bpw-h6-exl2 EXL2 5.0bpw 22,21 -> 43 GB 18/18 4+3+2+6 = 15/18
LoneStriker/lzlv_70b_fp16_hf-6.0bpw-h6-exl2 EXL2 6.0bpw > 48 GB TOO BIG
TheBloke/lzlv_70B-AWQ AWQ 4-bit OutOfMemory

My AI Workstation:

  • 2 GPUs (48 GB VRAM): Asus ROG STRIX RTX 3090 O24 Gaming White Edition (24 GB VRAM) + EVGA GeForce RTX 3090 FTW3 ULTRA GAMING (24 GB VRAM)
  • 13th Gen Intel Core i9-13900K (24 Cores, 8 Performance-Cores + 16 Efficient-Cores, 32 Threads, 3.0-5.8 GHz)
  • 128 GB DDR5 RAM (4x 32GB Kingston Fury Beast DDR5-6000 MHz) @ 4800 MHz ☹️
  • ASUS ProArt Z790 Creator WiFi
  • 1650W Thermaltake ToughPower GF3 Gen5
  • Windows 11 Pro 64-bit

Observations:

  • Scores = Number of correct answers to multiple choice questions of 1st test series (4 German data protection trainings) as usual
    • Primary Score = Number of correct answers after giving information
    • Secondary Score = Number of correct answers without giving information (blind)
  • Model’s official prompt format (Vicuna 1.1), Deterministic settings. Different quants still produce different outputs because of internal differences.
  • Speed is from koboldcpp-1.49’s stats, after a fresh start (no cache) with 3K of 4K context filled up already, with (+) or without (-) mmq option to --usecublas.
  • LoneStriker/lzlv_70b_fp16_hf-2.4bpw-h6-exl2: 2.4b-bit = BROKEN! Didn’t work at all, outputting only one word and repeating that ad infinitum.
  • LoneStriker/lzlv_70b_fp16_hf-2.6bpw-h6-exl2: 2.6-bit = FAIL! Achknowledged questions like information with just OK, didn’t answer unless prompted, and made mistakes despite given information.
  • Even EXL2 5.0bpw was surprisingly doing much worse than GGUF Q2_K.
  • AWQ just doesn’t work for me with oobabooga’s text-generation-webui, despite 2x 24 GB VRAM, it goes OOM. Allocation seems to be broken. Giving up on that format for now.
  • All versions consistently acknowledged all data input with “OK” and followed instructions to answer with just a single letter or more than just a single letter.
  • EXL2 isn’t entirely deterministic. Its author said speed is more important than determinism, and I agree, but the quality loss and non-determinism make it less suitable for model tests and comparisons.

Conclusion:

  • With AWQ not working and EXL2 delivering bad quality (secondary score dropped a lot!), I’ll stick to the GGUF format for further testing, for now at least.
  • Strange that bigger quants got more tokens per second than smaller ones, maybe that’s because of different responses, but Q4_K_M with mmq was fastest - so I’ll use that for future comparisons and tests.
  • For real-time uses like Voxta+VaM, EXL2 4-bit is better - it’s fast and accurate, yet not too big (need some of the VRAM for rendering the AI’s avatar in AR/VR). Feels almost as fast as unquantized Transfomers Mistral 7B, but much more accurate for function calling/action inference and summarization (it’s a 70B after all).

So these are my - quite unexpected - findings with this setup. Sharing them with you all and looking for feedback if anyone has done perplexity tests or other benchmarks between formats. Is EXL2 really such a tradeoff between speed and quality in general, or could that be a model-specific effect here?

Here’s a list of my previous model tests and comparisons or other related posts:

Disclaimer: Some kind soul recently asked me if they could tip me for my LLM reviews and advice, so I set up a Ko-fi page. While this may affect the priority/order of my tests, it will not change the results, I am incorruptible. Also consider tipping your favorite model creators, quantizers, or frontend/backend devs if you can afford to do so. They deserve it!

  • lone_striker@alien.topBEnglish
    1·
    1 year ago

    For the 2.4bpw and 2.6bpw exl2 models, you have to change a setting in ooba to get them to generate coherent text. Disable this setting:

    Add the bos_token to the beginning of prompts

    https://preview.redd.it/4v8m7ciu0y1c1.png?width=356&format=png&auto=webp&s=785837b8466a3bcda3e49477424b7c377a8d542f

    The very low bpw models need the above setting as well as being more strict with the prompt format. The higher bpw models are more flexible and can deal with prompt formats they were not specifically tuned for.

    I would also set the VRAM for 2.4 to use only a single GPU. Spreading them out over two GPUs is not needed and will slow them down. That’s the main reason I generate 2.4 (and 2.6bpw) versions is to allow people with only a single 3090 or 4090 to run 70B models at full speeds. Though obviously quality will be lower than the higher-bit models. For 2.6bpw to fit on a single 24 GB VRAM GPU, you will need to enable the cache_8bit option.

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      Does 8-bit cache reduce quality or speed or what’s the disadvantage of it? (If it had none, it would be default, I assume.)