I’m not really talking about a cryptocurrency, more of an anonymous collaborative system where your local copy of the blockchain syncs up to the most recent stage of the model’s training and dataset, and the process of mining is a pooled effort to find the next best loss reduction.

Training data would obviously be a problem, but I’m sure there are ways to append data to the dataset in the mining process.

Something decentralized, massive, and unstoppable.

Man I’m high…Time to go talk to Mistral.

  • __SlimeQ__@alien.topBEnglish
    1·
    1 year ago

    This isn’t an original concept but the data transfer rates make distributed training like this pretty impractical. Unless you’re doing some galaxy brained techniques you will need to distribute like 120gb every step (for a 70B model) and training on an old step is a waste of time. So parallelizing it on internet connected machines is a horrible option.

    You don’t really need this setup anyways as you can train a 4bit 13B Lora on a gaming pc. This would only be useful for big foundation models, maybe, in which case you’re going to get way faster results by just renting some cloud gpus

  • Tiny_Nobody6@alien.topBEnglish
    1·
    1 year ago

    IYH using u/stunspot AI bot Universal Evaluator gives 70/100 points

    As a Universal Evaluator with expertise in blockchain technology, I will evaluate the idea of using a blockchain-like system where the process of mining trains a decentralized model. I will use a 1-100 point scoring system to rate the merits of this idea.

    Scoring breakdown:

    1. Conceptualization (30 points): The idea of utilizing a blockchain-like system for training a decentralized model is intriguing. It combines the benefits of decentralized consensus and collaborative mining efforts to improve the model’s training process.

    ​

    1. Feasibility (40 points): The feasibility of this idea depends on several factors. Firstly, the technical implementation of integrating a decentralized model training process into a blockchain-like system needs to be carefully designed and implemented. Secondly, ensuring the scalability and efficiency of the system is crucial, as training a model requires significant computational resources. Finally, addressing the challenge of obtaining and appending training data in a decentralized manner is a critical aspect that needs further consideration.

    ​

    1. Potential Benefits (30 points): The proposed system has the potential to provide benefits such as enhanced security and immutability through the use of blockchain technology. It can also promote collaboration and incentivize participation through the mining process. Additionally, the decentralized nature of the system could make it resistant to censorship and single-point failures.

    ​

    Based on the evaluation, I would assign a score of 70 out of 100 to this idea.

    ​

    Improvement Recommendation:

    While the idea shows promise, there are some aspects that need further refinement. Addressing the challenge of obtaining and appending training data in a decentralized manner is a crucial consideration. Additionally, it would be beneficial to explore potential use cases and evaluate the scalability and efficiency of the proposed system. Conducting a thorough analysis of the potential risks and limitations associated with adopting this approach would also be valuable.

    ​

    In summary, the idea of using a blockchain-like system for training a decentralized model is intriguing, but it requires further development and considerations to ensure its feasibility and practicality.

    ​

    Should the idea be improved?

    Yes, the idea should be improved to address the challenges mentioned above and to further refine the concept. This will help enhance its feasibility and potential benefits.