IRT Dataset Generator

Dataset Generator for IRT Analysis

This script generates simulated datasets specifically designed for Item Response Theory (IRT) analysis, a robust framework commonly used in psychometrics.

IRT is a powerful method for modeling the relationship between latent traits (such as abilities or attitudes) and individual responses to test items or questionnaires (Embretson & Reise, 2000).

The generator focuses on the 2-Parameter Logistic (2PL) model, enabling the simulation of datasets that reflect a wide range of item characteristics and subject abilities, with support for various scenarios such as skewed distributions, polytomous items, and missing data.

Understanding IRT and the 2PL Model

IRT provides a modern approach to measuring latent traits by modeling the relationship between an individual’s ability and the characteristics of test items. The 2PL model is an extension of the Rasch model, also known as the 1-Parameter Logistic (1PL) model, by incorporating both item difficulty and item discrimination parameters. These parameters allow for more flexible modeling of how individuals with different ability levels interact with test items.

In the 2PL model, each item has two key parameters:

• Difficulty (\( b_i \)): Represents the level of the trait required to have a 50% probability of answering the item correctly. Higher values indicate more difficult items.
• Discrimination (\( a_i \)): Measures how well the item distinguishes between individuals with higher and lower abilities. Higher values indicate that the item is more effective at differentiating between individuals near the item’s difficulty level.

The probability of a correct response to an item is modeled by the 2PL formula:

\[ P(\text{correct}) = \frac{1}{1 + e^{-a_i(\theta - b_i)}} \]

where \( P(\text{correct}) \) is the probability of a correct response, \( \theta \) is the individual’s ability, \( a_i \) is the discrimination parameter, and \( b_i \) is the item’s difficulty.

Features of the Dataset Generator

The dataset generator implements the 2PL model across multiple scenarios, allowing users to simulate a variety of testing environments with different item and subject characteristics. It is highly customizable, supporting several key features that make it useful for researchers and practitioners conducting psychometric analysis:

1. Scenario-Specific Simulation

The generator includes a set of predefined scenarios, each reflecting different psychometric conditions. For instance, the 'homogeneous' scenario assumes items have relatively similar characteristics (narrow ranges of difficulty and discrimination), while the 'heterogeneous' scenario introduces a wider spread in item properties. Other scenarios model extreme difficulty, skewed distributions, and varying discrimination levels. This flexibility enables users to simulate diverse testing environments, including those that mimic real-world tests with varied item pools.

For example, in the 'highDiscrimination' scenario, items are designed to have high discrimination values, meaning they are more sensitive to small differences in ability, while the 'lowDiscrimination' scenario generates items that have less sensitivity, making it more difficult to distinguish between test-takers with different abilities.

2. Skewed Difficulty Distributions

The generator supports scenarios where item difficulties follow skewed distributions, allowing for the simulation of tests that may be intentionally designed to be easier or harder for specific populations. This feature is particularly useful for psychometricians aiming to develop tests with non-normal difficulty distributions, such as those targeting low-ability or high-ability populations. In scenarios with skewed difficulty (e.g., 'skewedDifficulty' or 'highlySkewedDiscrimination'), a transformation is applied to adjust the item difficulties based on a skewness parameter, ensuring that the items align with the desired distribution.

The skewness is applied mathematically, transforming the difficulty values according to the scenario's skewness factor. For positive skewness, more items are concentrated at the easier end of the spectrum, while for negative skewness, more items are challenging.

3. Polytomous Item Scoring (More Than Two Categories)

In addition to simulating binary item responses (correct/incorrect), the generator also supports polytomous items—those with multiple response categories. This is essential for tests that score responses along a graded scale, such as rating scales or partial credit items. When simulating polytomous items, the generator calculates the probabilities of achieving each score category based on the individual’s ability (\( \theta \)) and the item’s difficulty and discrimination parameters. The number of score categories is controlled by the parameter \( k \), and the probability of each score is derived from the 2PL model.

For polytomous items, the score is determined by comparing random draws against the cumulative probabilities of scoring in each category, reflecting real-world testing where responses may fall into different levels rather than simply correct or incorrect.

4. Missing Data Simulation

The generator includes functionality to simulate datasets with missing data, which is a common feature in real-world testing scenarios where not all participants answer every question. Missing data is introduced based on a specified missing rate. This capability is valuable for researchers who need to explore how missing responses affect IRT model estimates or wish to develop strategies for handling incomplete data. Future versions will expand on this feature by introducing more sophisticated methods for simulating non-random missing data patterns.

5. Generation of Trait Levels (Theta)

The generator simulates subject abilities, represented as \( \theta \), using a normal distribution with specified mean and standard deviation. To prevent extreme outliers, \( \theta \) values are constrained within reasonable bounds (e.g., between -6 and 6). These boundaries ensure that simulated datasets reflect realistic ability distributions, and are particularly useful when generating large samples of test-takers where extreme values could unduly influence model outcomes.

The ability level generation also supports fine-tuning for various populations by adjusting the mean and standard deviation of the trait level distribution.

Mathematical Formulas Implemented

The dataset generator is based on several mathematical formulas that define how item parameters and subject abilities are generated:

1. Item Difficulty Calculation:

\[ b_i = \mu_{b} + (R - 0.5) \times 2 \times \sigma_{b} \]

where \( b_i \) is the difficulty of item \( i \), \( \mu_{b} \) is the mean difficulty, \( R \) is a random value from a uniform distribution, and \( \sigma_{b} \) is the standard deviation of difficulty. Skewness transformations are applied for certain scenarios.

2. Discrimination Sampling:

\[ a_i = \exp\left(\ln(\mu_{a}) - 0.5 \times \ln\left(1 + \frac{\sigma_{a}^2}{\mu_{a}^2}\right) + Z \times \sqrt{\ln\left(1 + \frac{\sigma_{a}^2}{\mu_{a}^2}\right)}\right) \]

where \( a_i \) is the discrimination of item \( i \), \( \mu_{a} \) is the mean discrimination, \( \sigma_{a} \) is the standard deviation of discrimination, and \( Z \) is a sample from a normal distribution. Discrimination is constrained by scenario-specific minimum and maximum values.

3. Probability of Correct Response (2PL Model):

\[ P_i(\theta) = \frac{1}{1 + \exp(-1.702 \times a_i \times (\theta - b_i))} \]

where \( P_i(\theta) \) is the probability of a correct response to item \( i \) for a subject with trait level \( \theta \), \( a_i \) is the item’s discrimination, and \( b_i \) is the item’s difficulty.

4. Trait Level (Theta) Generation:

\[ \theta = Z \times \sigma_{\theta} + \mu_{\theta} \]

where \( \theta \) is the trait level of a subject, \( Z \) is a sample from a standard normal distribution, \( \sigma_{\theta} \) is the standard deviation of the trait, and \( \mu_{\theta} \) is the mean trait level. Theta is constrained within user-specified bounds.

5. Score Generation for Polytomous Items (with \( k \) categories):

\[ \text{score} = \min\left(\sum_{m=1}^{k-1} \mathbf{1}\left[ U < \sum_{n=1}^{m} P_{i,n}(\theta) \right], k-1 \right) \]

where \( \text{score} \) is the item score, \( \mathbf{1} \) is the indicator function, \( U \) is a random uniform variable, \( P_{i,n}(\theta) \) is the probability of scoring in category \( n \) for item \( i \) given \( \theta \), and \( k \) is the number of score categories.

Conclusion

This dataset generator provides a versatile tool for simulating IRT datasets across various scenarios, reflecting real-world psychometric testing conditions. With its support for skewed distributions, polytomous items, missing data, and customizable trait levels, it offers a powerful resource for researchers and psychometricians seeking to explore the behavior of IRT models under different conditions.

References

Baker, F. B. (2001). The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation.

Birnbaum, A. (1968). Some Latent Trait Models and Their Use in Inferring an Examinee's Ability. In F. M. Lord & M. R. Novick, Statistical Theories of Mental Test Scores. Addison-Wesley.

Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Lawrence Erlbaum Associates.

Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.