With high-end Android phones now packing upwards of 24GB of RAM, I think there’s huge potential for an app like this. It would be amazing to have something as powerful as the future Mistral 13B model running natively on smartphones!
You could interact with it privately without an internet connection. The convenience and capabilities would be incredible.
It’s not just RAM, you also need the processing power. Phones can’t do *good* LLMs yet.
If you watch the chatGPT voice chat mode closely on android, what it does is listen, with a local voice model (whisper.cpp), and then answers generally/quickly LOCALLY, for the first response/paragraph. While that’s happening, it’s sending what you asked to the servers, where the real text processing takes place. By the time your phone has run the simple local model and gotten a simple sentence for the first response and read that to you, it has MOSTLY gotten the full paragraphs of text back from the server and can read that. Even then, you still notice a slight delay.