Jouve's Randomized Reliability Estimation (JRRE) Calculator

Jouve's Randomized Reliability Estimation (JRRE)

Jouve's Randomized Reliability Estimation (JRRE) is a method aimed at improving the accuracy of test reliability estimates by averaging reliability measures over multiple randomized test splits. It enhances traditional split-half reliability methods by utilizing the Fisher-Yates shuffle algorithm to randomly permute test items, allowing for a statistically robust estimate of overall reliability. The JRRE method reduces dependency on a single arbitrary split by incorporating multiple test splits.

Steps in JRRE Method

1. Randomization of Test Items: The Fisher-Yates shuffle algorithm is employed to randomly permute the test items. This ensures that every permutation of the test items is equally probable.
2. Splitting the Test: After each shuffle, the test is divided into two halves.
3. Score Calculation: For each subject, the total score for both halves of the test is computed.
4. Pearson Correlation Coefficient Calculation: The Pearson product-moment correlation coefficient is calculated between the scores of the two halves.
5. Spearman-Brown Prophecy Formula: The Spearman-Brown formula is applied to the correlation coefficient to project the reliability of the full test, adjusting for the reduction in test length.
6. Final Reliability Estimation: The reliability estimates from all splits are averaged to compute the final JRRE value, representing the overall reliability of the test.

Fisher-Yates Shuffle Algorithm

The Fisher-Yates algorithm (Fisher & Yates, 1948) is used to randomly permute the array of test items, ensuring each permutation has equal probability. The algorithm proceeds as follows:

1. For each index \(i\), where \(i\) decreases from \(n-1\) to \(1\), pick a random index \(j\) such that \(0 \leq j \leq i\).
2. Swap the elements \(a[i]\) and \(a[j]\), where \(a\) is the array of items.

The resulting array is a random permutation of the original array. This algorithm operates in \(O(n)\) time, where \(n\) is the number of items, ensuring computational efficiency even for large datasets.

Pearson Correlation Coefficient

After the test is split into two halves, the Pearson correlation coefficient \(r\) is computed between the two sets of scores. The Pearson correlation coefficient measures the linear relationship between the two sets of data. For two sets of scores \(X = \{X_1, X_2, \dots, X_n\}\) and \(Y = \{Y_1, Y_2, \dots, Y_n\}\), the Pearson correlation coefficient is defined as:

\[ r = \frac{\sum_{i=1}^{n} (X_i - \overline{X})(Y_i - \overline{Y})}{\sqrt{\sum_{i=1}^{n} (X_i - \overline{X})^2} \sqrt{\sum_{i=1}^{n} (Y_i - \overline{Y})^2}} \]

where:

• \(n\) is the number of subjects.
• \(X_i\) and \(Y_i\) are the scores of the \(i\)-th subject on the two halves of the test, respectively.
• \(\overline{X}\) is the mean of the first set of scores, calculated as:
\[ \overline{X} = \frac{1}{n} \sum_{i=1}^{n} X_i \]
• \(\overline{Y}\) is the mean of the second set of scores, calculated as:
\[ \overline{Y} = \frac{1}{n} \sum_{i=1}^{n} Y_i \]
• The term \(\sum_{i=1}^{n} (X_i - \overline{X}) (Y_i - \overline{Y})\) represents the covariance between the two sets of scores.
• The terms \(\sum_{i=1}^{n} (X_i - \overline{X})^2\) and \(\sum_{i=1}^{n} (Y_i - \overline{Y})^2\) are the variances of the two sets of scores, and their square roots provide the standard deviations of the two distributions.

This formula quantifies the degree of linear association between the two halves, with values of \(r\) ranging from \(-1\) to \(1\), where \(r = 1\) indicates perfect positive correlation and \(r = 0\) indicates no correlation.

Spearman-Brown Prophecy Formula

Once the Pearson correlation coefficient \(r\) has been computed, the Spearman-Brown formula is applied to adjust for the fact that the test length has been halved. The formula for the adjusted reliability is given by:

\[ \text{Reliability}_{split} = \frac{2r}{1 + r} \]

where:

• \(r\) is the Pearson correlation coefficient between the two halves.

The Spearman-Brown formula extrapolates the reliability of the full-length test from the correlation between the two halves. This adjustment is necessary because reliability typically increases with test length.

Averaging Reliability Across Splits

The final JRRE value is computed by averaging the reliability estimates over multiple random splits. If \(N\) random splits are performed, the JRRE is given by:

\[ JRRE = \frac{1}{N} \sum_{i=1}^{N} \text{Reliability}_{split_i} \]

where:

• \(N\) is the number of random splits.
• \(\text{Reliability}_{split_i}\) is the reliability for the \(i\)-th random split, calculated using the Spearman-Brown formula:
\[ \text{Reliability}_{split_i} = \frac{2r_i}{1 + r_i} \]
• \(r_i\) is the Pearson correlation coefficient for the \(i\)-th split, calculated as described previously.

The JRRE process reduces the variability in reliability estimation by averaging over multiple splits, thereby mitigating the influence of any single partition. The final estimate provides a robust measure of the test’s internal consistency.

Convergence and Sensitivity

The JRRE computation typically involves running a large number of iterations to ensure the reliability estimate stabilizes. In practical implementation, the JRRE is computed over a large number of iterations to ensure that changes in the average reliability across iterations become negligible. Convergence is monitored by tracking the change in reliability estimates between iterations, and the algorithm halts when the change falls below a pre-specified threshold (e.g., \(1 \times 10^{-9}\)).

Mathematical Considerations and Applications

The JRRE method offers a mathematically rigorous approach to estimating test reliability, particularly in the following scenarios:

• Small Sample Sizes: JRRE is robust when applied to small sample sizes, where traditional split-half methods may produce unstable results. By averaging across multiple splits, JRRE reduces the impact of random variability and provides a more stable estimate.
• Heterogeneous Item Sets: In tests with items of varying difficulty or discrimination, JRRE accommodates the inherent variability by using multiple random splits, resulting in a more accurate reliability estimate.
• High-Stakes Testing: In contexts where test reliability is critical, such as certification exams or psychological assessments, JRRE offers a more precise and less arbitrary reliability estimate than traditional methods.

Mathematically, JRRE can be viewed as a Monte Carlo approach to reliability estimation, where the random splits serve as independent samples. The Fisher-Yates shuffle ensures unbiased randomization, while the Spearman-Brown formula ensures that the reliability estimates are properly adjusted for the full test length. By averaging over a large number of splits, JRRE provides a reliable estimate of the test's internal consistency, with reduced variance compared to single-split methods.

References

Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6(4), 475-494. https://doi.org/10.1177/001316444600600405

Fisher, R. A., & Yates, F. (1948). Statistical tables for biological, agricultural and medical research (3rd ed.). Oliver & Boyd.