I am talking about this particular model:

https://huggingface.co/TheBloke/goliath-120b-GGUF

I specifically use: goliath-120b.Q4_K_M.gguf

I can run it on runpod.io on this A100 instance with “humane” speed, but it is way too slow for creating long form text.

https://preview.redd.it/fz28iycv860c1.png?width=350&format=png&auto=webp&s=cd034b6fb6fe80f209f5e6d5278206fd714a1b10

These are my settings in text-generation-webui:

https://preview.redd.it/vw53pc33960c1.png?width=833&format=png&auto=webp&s=0fccbeac0994447cf7b7462f65d79f2e8f8f1969

Any advice? Thanks

  • kruk2@alien.topBEnglish
    1·
    1 year ago

    or open the UI, go to model page, right click on the layers slider -> inspect element
    and update max value for the input field from 128 to 256

      • MINIMAN10001@alien.topBEnglish
        1·
        1 year ago

        I mean it makes sense The value is chosen we’re simply chosen for being a reasonable window at the time.

        There was nothing hard coded about them they were simply a range of values that they had set for the UI.

        It certainly is interesting though.