Exercise 30 - Reliability of a Rating Scale: Test-Retest Reliability Coefficient and Coefficient of Determination Statistical Guide
To review correlation coefficients, see the Statistical Guide for Exercise 24. To review coefficients of determination and percentage of explained variance, see the Statistical Guide for Exercise 25.
To determine test-retest reliability, the test is administered to one group of examinees twice (often with a week or two intervening between the two administrations of the test). This yields two scores for each examinee. Computing the correlation coefficient for the relationship between the two sets of scores indicates the stability of the scores over time. For instance, a high correlation coefficient indicates that those who scored high on the first administration also scored high on the second administration. Likewise, those who scored low on the first administration also scored low on the second administration. When the correlation coefficient is used to express the stability of test scores over time, the coefficient is commonly referred to as the test-retest reliability coefficient. Despite the different names, the test-retest reliability coefficient is mathematically the same as the correlation coefficient.
Excerpt from the Research Article:
We assessed the reliability of the Behavioral and Emotional Rating Scale Second Edition: Parent Rating Scale (BERS-2; PRS). All parents of kindergarteners through second graders at the elementary school (N=122) and sixth graders at the middle school (N=169) received a mailed copy of the PRS. Parents were informed that upon receipt of the first protocol, a second PRS would be sent to them that was also filled out and returned. A return rate of 26.89% was achieved.
Assessments like the PRS, which may be used for screening, planning, and evaluation purposes and for which data are reported individually, should have reliability coefficients of at least 0.80.