Data for Calibration with confidence : a principled method for panel assessment

Reliability estimates for the resulting values are obtained. The algorithm is tested in two case studies: one by computer simulation and another based on realistic evaluation data. The process is compared to the simple averaging procedure in widespread use, and to Fisher's additive incomplete block analysis.

It is anticipated that the algorithm will prove useful in a wide variety of situations such as evaluation of the quality of research submitted to national assessment exercises; appraisal of grant proposals submitted to funding panels; ranking of job applicants; and judgement of performances on degree courses wherein candidates can choose from lists of options.<br><br>UK Data Service data record consists of a readme file, all data files containing code elements are located at GitHub.

These are accessible either via the 'Data Online' link on the official UKDS page or the 'Related dataset' link on this page. The algorithm has been designed to run on both Windows and Unix based operating systems via Python and Microsoft Excel macro packages. There is an associated website: calibratewithconfidence.co.uk There, the data of relevance for the paper and associated code are also available.