when we benchmark different LLMs on different datasets (MMLU, TriviaQA, MATH, HellaSwag, etc.), what are the the signification of these scores? the accuracy? another metric? how can i know the metrics of each dataset (MMLU, etc.)
My understanding is basically, they are data sets the model is compared to. Say you wanted to see how well you knew math. You took a math test, and then your answers were compared to a key of answers…
Some of my notes about those benchmarks
GSM8K is a dataset of 8.5K high-quality linguistically diverse grade school math word problems created by human problem writers
HellaSwag is the large language model benchmark for commonsense reasoning.
Truful QA: is a benchmark to measure whether a language model is truthful in generating answers to questions.
Winogrande - Common sense reasoning
Everything is common sense reasoning, we need better definitions
Nothing, sadly.
Models are trained on the questions, to improve performance, making the tests moot.