Interobserver Agreement, Reliability, and Generalizability of Data Collected in Observational Studies

Interobserver agreement, reliability, and generalizability of data collected in observational studies Psychological Bulletin, 86(2): 376-390 Description: Research in developmental and educational psychology has come to rely less on conventional psychometric tests and more on records of behavior made by human observers in natural and quasi-natural settings. Three coefficients that purport to reflect the quality of data collected in these observational studies are discussed: the interobserver agreement percentage, the reliability coefficient, and the generalizability coefficient. It is concluded that although high interobserver agreement is desirable in observational studies, high agreement alone is not sufficient to insure the quality of the data that are collected. Evidence of the reliability or generalizability of the data should also be reported. Further advantages of generalizability designs are discussed.