Today the judgment of a group of experts is often at the centre of the decision making process when a problem is deemed difficult to quantify. Whenever one or more objects are evaluated by two or more observers (i.e. judges, experts) we must pose the question as to what extent we can rely on that evaluation. There are many sources of reliability of such group judgments, but the final tool of verification should be the degree of agreement among experts. If the evaluated objects are listed in order of their significance, one of the rank correlation methods is often applied to this end. However, rank correlation methods have their limitations, which are discussed in the first part of the presentation. Next, the case is presented where the preferences of observers are expressed by using an ordinal scale, but the objects are not ranked. When we have a set of grades for the object we can apply many measurements to verify the agreement. Nevertheless, what does the agreement between opinions on the same object really mean, and what type of dispersion measurements would be the best reflection of that agreement? We provide the answer, which is based on modules between grades, and which allows us to develop an appropriate coefficient of agreement. We also indicate the basis of calculating the actual distribution of the coefficient in order to test the statistical significance of the measurement.
|Keywords:||Subjective Judgment, Group Judgment, Ordinal Scale, Rank Correlation Methods, Coefficient of Concordance, Coefficient of Agreement|
Assistant Professor, Department of Management Process, Cracow University of Economics, Krakow, Poland
There are currently no reviews of this product.Write a Review