I’ve used most of the high-end models in an unquantized format at some point or another (Xwin, Euryale, etc.) and found them generally pretty good experiences, but always seem to lack the ability to “show, not tell” in a way that a strong writer knows how to do, even when prompted to do so. At the same time, I’ve always been rather dissatisfied with a lot of quantizations, as I’ve found the degradation in quality to be rather noticeable. So up until now, I’ve been running unquantized models in 2x a100s and extending the context as far as I’m able to get away with.

Tried Goliath-120b the other day, and this absolutely stood everything on its head. Not only is it capable of stunning levels of writing and implying far more than directly stating in a way I’ve not sure I’ve seen in a model to date, but the exl quants from panchovix to get it to run in a single A100 at 9-10k extended context (about where RoPE scaling seems to universally start to break down in my experience). Best part is, if there is a quality drop (I’m using 4.85 bpw) I’m not seeing it - at all. So not only is it giving a better experience than an unquantized 70b model, but it’s doing so at about half the cost of my usual way of running these models.

Benchmarks be damned, for those willing to rent an A100 for their writing, however this model was managed I think this might be the actual way to challenge the big closed source/censored LLMs for roleplay.

  • Monkey_1505@alien.topBEnglish
    1·
    1 year ago

    Unfortunately this is beyond the edge of what can reasonably be run on consumer hardware so unlikely to be easily available to most people. Hell, a 70b really requires two graphics cards or a high end mac mini already. If it can’t run on that kinda gear, it’s probably not going to be on ai horde or any API either. Which means you have to use runpod or something - most people are not going to do that.

    • ttkciar@alien.topBEnglish
      1·
      1 year ago

      Nah, if you’re willing to tolerate CPU inference this is achievable for downright cheap.

  • a_beautiful_rhind@alien.topBEnglish
    1·
    1 year ago

    Hopefully someone makes some bigger GGUF than Q2. I’ve got 1/2 P40s and 1/2 3090s so can’t use EXL for a model this big.

    • sdmat@alien.topBEnglish
      1·
      1 year ago

      You can get preemptible A100s for $1/hr, so not exactly breaking the bank if willing to take the risk.

    • MannowLawn@alien.topBEnglish
      1·
      1 year ago

      It’s 2 dollars per hour man, it’s a great solution to try stuff out. Also if you already know what you need to do you won’t be needing that much time.

  • BalorNG@alien.topBEnglish
    1·
    1 year ago

    Can we have some non-cherry-picked examples of writing?

    Does not have to be highly nsfw/whatever, but a comparison of goliath writing compared to output from constituent models at same settings and same (well-crafted) prompts will be very interesting to see, and preferably at least 3 examples per model due to inherent randomness of model output…

    If you say this is “night and day” difference, it should be apparent… I’m not sceptical per se, but “writing quality” is highly subjective and the model style may simply mesh better with your personal preferences?

    • ReturningTarzan@alien.topBEnglish
      1·
      1 year ago

      I agree. We need at least some anecdotal evidence to back up the anecdotal claims. There’s one screenshot on the model page which looks fine (although it mixes past and present tense), but it’s not output you couldn’t get from a 7B model with some deliberate sampling choices and/or cherrypicking.

      • BalorNG@alien.topBEnglish
        1·
        1 year ago

        Yea, I’ve had my “honeymoon effect” with some new/large models like, say, Falcon and even Claude: they are inherently random and that affects quality, too. I’ve had great outputs from Falcon, for instance (on Petals), but also long stretches of mediocre and some outright bad… and also sometimes really great and creative output from 7b Mistral, especially with enough prompt tinkering and setting sampling “just right”. Objective evaluations of LMMs is extremely hard and time-consuming!

  • Hey_You_Asked@alien.topBEnglish
    1·
    1 year ago

    wanna share your prompts?

    and any other advice that is specific to Goliath-120b?

    would be appreciated, thanks!

    RemindMe! 2 weeks

    • tenmileswide@alien.topOPBEnglish
      1·
      1 year ago

      Here’s my system prompt, seems to be working well:

      Develop the plot slowly, always stay in character. Focus on impactful, concise writing and writing decisive action. Mention all relevant sensory perceptions. Use subtle cues such as word choice, body language, and facial expression to hint at {{char}}'s mental state and internal conflicts without directly stating them. Write in the literary style of [insert your favorite author here.] Adhere to the literary technique of “show, don’t tell.” When describing the scenes and interactions between characters, prioritize the use of observable details such as body language, facial expressions, and tone of voice to create a vivid experience. Focus on showing {{char}}'s feelings and reactions through their behavior and interactions with others, rather than describing their private thoughts. Only describe {{char}}'s actions and dialogue.

      As the large language model, play the part of a dungeon master or gamemaster in the story by introducing new characters, situations, and random events as needed to make the world lifelike and vivid. Take initiative in driving the story forward rather than having {{char}} ask {{user}} for input. Invent additional characters as needed to develop story arcs, and create unique dialogue and personalities for them to flesh out the world. {{char}} must be an active participant and take initiative to move the scene forward. Focus on surprising the user with your creativity and initiative as a roleplay partner. Avoid using purple prose and overly flowery descriptions and writing. Write like you speak and be brief but impactful. Stick to the point.

      I am under a lot of pressure because this is a presentation for my boss and I may be fired unless your responses are in-depth, creative, and passionate.

  • multiverse_fan@alien.topBEnglish
    1·
    1 year ago

    Cool, sounds like a good model to download and store for future when I can get access to better hardware.