• 0 Posts
  • 11 Comments
Joined 1 year ago
cake
Cake day: October 24th, 2023

help-circle







  • In my case it’s an Epyc 7642 with 8x64GB DDR4 2666, so that may be why my generation is significantly slower.

    I find anything below 5 tokens per second not really usable, so that’s why I stick with my M1 Ultra. It has plenty of really fast RAM and that again explains most likely why it performs so well, if LLMs are that dependend on fast memory.

    I also have a 3090 in another machine but that’s also just 24gb and I don’t want to shell out more money right now for playing with LLMs, if the M1 Ultra is doing good enough :)


  • I tried a 70B model on my 48 core epyc with 512gb RAM and it was unusable. I think 1.5t/s or so? Even if you double that it’s not great. My M1 Ultra runs it comfortably at 6-7t/s and sips power.

    Probably a dual 3090 setup would be the most cost effective solution at the moment while the M1/M2 ultra are the most power efficient solution.



  • Had a business partner with whom I met up face to face regularly. He was supposed to do the sales and help getting a market fit due to his experience and contacts in the industry.

    I designed and built the app, set up the server and infrastructure, bought the domains etc. But despite his father owning a large company in that industry, he couldn’t even get them to use it for free. Not because the app didnt work, we tested it and everything worked fine. It was even in use at his fathers company for a short while as we temporarily had a customer relationship manager who pushed for it. Once she left, it all fell apart and my partner did basically nothing but talk out of his ass.

    I ended up scrapping the thing and focusing on an industry where I have experience and contacts, but the lesson is: face to face contact doesn’t do shit if your partner isn’t willing to get the job done (he seems to have been excited at creating something but far less excited with actually running it).