Introducing the Artificial Analysis Decision Index v2 – LinkedIn

Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from Nace AI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 ( Fastino Labs ) and Mammoth 27B ( Base Labs ) Updates in Decision Index v2: ➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half ➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right ➤ AA-Symmetry (15%): This evaluates whether models change their answers for equivalent questions
Read More