what do us poor folks on Linux do?

Run llama.cpp and any of the models listed here, that stuff has been around for months.

TheBloke has a lot of models converted to GGUF format which you need for llama.cpp.

Quick Start Guide (requires Nix, otherwise compile llama.cpp manually):

$ GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/TheBloke/guanaco-7B-GGUF
$ cd guanaco-7B-GGUF
$ git lfs pull --include=Guanaco-7B.Q4_0.gguf
$ nix run github:ggerganov/llama.cpp -- -m Guanaco-7B.Q4_0.gguf --instruct
> Write haiku about a penguin
 A penguin walks on ice,
 Takes a plunge in the sea,
 Hides his feet from me!

KoboldAI has an “KoboldGPT” mode, which is for answering questions. You can try it at https://lite.koboldai.net/ or run it on your own.

Here’s an example conversation I just had with it (emoji icons added):

🤖 Hello, I am KoboldGPT, your personal AI assistant. What would you like to know?

🧑 What is the name of the original version of the game Monopoly?

🤖 The original version of the game Monopoly is called “The Landlord’s Game.” It was created by Elizabeth Magie in the early 20th century. However, the most well-known version of Monopoly today is based on a variant called “Monopoly” that was designed by Charles Darrow in 1935.

🧑 Did “The Landlord’s Game” have any purpose besides being a board game?

🤖 Yes, “The Landlord’s Game” had a specific purpose beyond being a board game. Elizabeth Magie created it as a way to illustrate the negative aspects of concentrating land in private monopolies. She believed that it could be used as a tool for teaching economic concepts such as rent, taxes, and the distribution of wealth.

🧑 What economic ideology was associated with this game?

🤖 Elizabeth Magie was influenced by Henry George’s philosophy of Georgism. Georgists believe that land should belong to all people equally, and that taxation should be based on the value of land rather than on income or production.

i don’t know a single thing about whatever these things your talking about are, but i just want to say, you’re the first person i’ve ever seen suggest that edge isn’t a joke.

As others said you can host Koboldai on your own, and if you don’t have a powerful GPU, anyone can use powerful llm models via the AI horde

Host it locally https://github.com/nomic-ai/gpt4all

man -k

OpenAssistant

I think it’s far more likely that easier solutions to hosting your own LLM/ChatGPT/etc will be pushed, rather than someone (or a group of someones) offering and maintaining it - as you mentioned, that takes a lot of resources. Think about the issues that various Fediverse software has had in terms of keeping up with the load sometimes, and LLMs use way more than most Fediverse applications. Especially if you want it to be as snappy as the current ones already out there.

oogabooga is an option for self-hosted LLMs

https://github.com/oobabooga/text-generation-webui

Chatbot arena ranks LLM, including some truly open source ones (apache license)

Regarding UI, maybe look here?

Maybe something like this? https://huggingface.co/ https://huggingface.co/chat/

The model seems to be based on the Llama model (but not only, there seem to be multiple models). Tho I didn’t really look into it on how they get their database. It also has more hallucinations than chatgpt.

deleted by creator

deleted by creator

At work today, I just heard the term, “Federated Machine Learning” - where individual corporations keep their proprietary learning data, but make the models shareable. I wonder if something similar could be done with activitypub?

(Feel free to steal this idea if it’s worth stealing)

I installed Edge on Linux because it’s fun watching Linux people get angry about it