Hi. So I am a bit new to NLP and ML as a whole and I am looking to create a text classification model. I have tried it with deBERTa and the results are decent(about 70%) but I need more accuracy. Are Generstive models a better alternative or should I stick to smaller models like Bert or maybe even non-NN classifiers and work on better dataset quality?

  • phree_radical@alien.topBEnglish
    1·
    1 year ago

    Maybe it’s overkill, idk, but if you want higher accuracy, it’s an option

    You can just list examples from your dataset and let the LLM complete the last one

    Example:

    # Classify text

(a) advertisement
(b) poetry
(c) information

Ignore real-time Al and customers will do the same to you. Our vector database is AI-ready and proven at scale.
Class: (a)

I find no peace, and all my war is done. I fear and hope. I burn and freeze like ice. I fly above the wind, yet can I not arise
Class: (b)

YOUR BEST COMES OUT OF THE BLUE. EXPLORE BOISE STATE
Class: (a)

Two-month ramp closure: northbound OR 99W onto OR 217 north Starts May 31 of Transportation Oregon Department OR 217 AUXILIARY LANES
Class: (c)

Staying healthy. Staying active. We have it all right here. IN YOUR PRIME LEARN MORE LIVING YOUR BEST LIFE
Class: (a)

Go further, FASTER. Take the world's premier English- proficiency test in less than 2 hours!
Class: (a)

A rhinoceros beetle is a living thing. Rhinoceros beetles grow and respond to their environment. They need food and water.
Class: (c)

Our vice runs beyond all that old men saw, And far authentically above our laws, And scorning virtues safe and golden mean, Sits uncontrolled upon the high extreme.
Class: (b)

{your text here}
Class: ({generate one token}

    I don’t know about the task you have in mind specifically, but you can do just about anything with a 13B llama model. Picking a fine-tune doesn’t matter if you use examples instead of instructions. 7B Mistral seems to do fine with this example (even GPT2 can do some classification), but in-context learning is remarkably better at 13B, picking up a lot more nuance

    • Shoddy_Vegetable_115@alien.topOPBEnglish
      1·
      1 year ago

      My classification task is to classify a given essay into AI generated and human generated. And I need the answer to be between 0 and 1(both included) with 1 being AI generated and 0 being human generated.

      Few-shot examples is a good idea for most classification tasks but I don’t think Generative LLMs can understand the more intricate semantic patterns to differentiate between the AI and human generated with just a few examples but I’ll try it once and let you know!

      Btw do you think fine-tuning would be better?

    • sshh12@alien.topBEnglish
      1·
      1 year ago

      +1, when in doubt, LLM it out.

      You could also ask for explanations so when it gets it wrong, you can work on modifying your prompts/examples to get better performance.

      Potentially you wouldn’t want to do this if:

      • Your classification problem is very unusual/cannot be explained by a prompt
      • You want to be able to run this extremely fast or on a ton of data
      • You want to learn non-LLM deep learning/NLP (in which case I would’ve suggested basically some form of finetuning BERT)