ai governance and threat modelling

November 9, 2023 · 86 words · 1 min read · tech, privacy

how are ml systems measured.

While training this is the loss and accuracy of the prediction.

For LLMs there are benchmarks like,

• ARC (A12 Reasoning Challenge)

  

  • Knowledge Reasoning framework, instead of a Q&A like SQuAD (Stanford Question Answering Dataset) and (SNLI) Stanford Natural Language Inference.
  • Distributed evidence in sentences, meaning that the answer is evenly distributed throughout the question.

• API-bank, LLM tool usage, evaluation

  

  .

• hellaSwag, to evaluate common sense of LLMs by having “adversarial endings”

• GLUE (General Language Understanding) and SuperGLUE, MMLU,