Date of Award
1-1-2011
Document Type
Campus Access Dissertation
Department
Educational Studies
Sub-Department
Educational Psychology / Research
First Advisor
Robert Johnson
Abstract
Inter-rater reliability coefficients are often reported in performance assessment as a measure of rating quality. Rater effects, as a systematic error in rating, may change the decision made about an examinee when an absolute interpretation of scores, such as a pass/fail decision, is used. Little research, however, examines the robustness of different inter-rater reliability estimators to rater effects. The purpose of the present research is to inform assessment practitioners or researchers reporting inter-rater reliability of (a) the more appropriate estimator in the presence of rater effects to achieve the most accurate estimates of inter-rater reliability and (b) the possible biases of the estimators under various conditions, such as number of scale points, initial inter-rater reliability, and number of papers. Results from this study indicate that the leniency/severity effects and the number of scale points substantially affect the accuracy of different inter-rater reliability estimators. The phi coefficient (or index of dependability) is not always the most accurate estimator, and it becomes the best choice only when the leniency/severity effect is large and the number of scale points is large. However, for other estimators, the increase in the number of scale points does not necessarily help the improvement of estimation accuracy.
Rights
© 2011, Min Zhu
Recommended Citation
Zhu, M.(2011). Robustness of Inter-Rater Reliability Estimators to Rater Leniency/Severity Effects In An Absolute Decision Setting - A Monte Carlo Study of Performance Ratings. (Doctoral dissertation). Retrieved from https://scholarcommons.sc.edu/etd/1040