The Reliability of Group Judgment: Some Remarks on the Measurement of Agreement among Experts

By Pawel Cabala.

Published by The Social Sciences Collection

Format Price
Article: Print $US10.00
Article: Electronic $US5.00

Today the judgment of a group of experts is often at the centre of the decision making process when a problem is deemed difficult to quantify. Whenever one or more objects are evaluated by two or more observers (i.e. judges, experts) we must pose the question as to what extent we can rely on that evaluation. There are many sources of reliability of such group judgments, but the final tool of verification should be the degree of agreement among experts. If the evaluated objects are listed in order of their significance, one of the rank correlation methods is often applied to this end. However, rank correlation methods have their limitations, which are discussed in the first part of the presentation. Next, the case is presented where the preferences of observers are expressed by using an ordinal scale, but the objects are not ranked. When we have a set of grades for the object we can apply many measurements to verify the agreement. Nevertheless, what does the agreement between opinions on the same object really mean, and what type of dispersion measurements would be the best reflection of that agreement? We provide the answer, which is based on modules between grades, and which allows us to develop an appropriate coefficient of agreement. We also indicate the basis of calculating the actual distribution of the coefficient in order to test the statistical significance of the measurement.

Keywords: Subjective Judgment, Group Judgment, Ordinal Scale, Rank Correlation Methods, Coefficient of Concordance, Coefficient of Agreement

International Journal of Interdisciplinary Social Sciences, Volume 5, Issue 2, pp.409-424. Article: Print (Spiral Bound). Article: Electronic (PDF File; 875.685KB).

Dr. Pawel Cabala

Assistant Professor, Department of Management Process, Cracow University of Economics, Krakow, Poland

I have been a university professor at Cracow University of Economics since 1995. I am currently teaching business management and researching such subjects as business planning, the theory of managerial decision-making, and the theory of human action (praxeology). I earned my M.A. in Marketing from the CUE in 1994 after which time I worked in a Polish investment bank for two years. I completed a Ph.D. in Business Administration from the CUE in 2003. In addition, I gained an M.A. in Philosophy from Jagiellonian University in 2002. I am also a business consultant. My interests in work and research are strategic management, organizational behavior, game theory, statistics and philosophy of action. Outside of work, I enjoy listening to music and spending time with my family in the Polish mountains.


There are currently no reviews of this product.

Write a Review