Effect of measurement scales on results of item response theory models and multivariate statistical techniques

Abstract

The study investigates the effects of response scales of items on results of item response theory (IRT) models and multivariate statistical techniques. A total of sixty-four datasets have been simulated under various conditions such as item response format, number of dimensions underlying response scales, and sample size using R package MIRT command: simdata (a, d, N, itemtype). Two main statistical techniques - IRT models and Factor Analysis - are employed in analysing the simulated datasets using standard R 3.4.3 codes. We find that there is a direct relationship between parameters of IRT and those of factor models, particularly item discrimination and factor loadings. The results also show that the overall fitness of the item response model increases with increasing scale points for higher dimensionality and sample size 150 and higher. The fitness deteriorates over increasing scale points for small sample sizes for unidimensional IRT model. Again, the number of influential indicators on factors increases with increasing scale-points, which improves the fitness of the model. The results indicate that unrealistic factor solution may be obtained if we attempt to extract higher factor solution than the underlying dimensionality on few scale-points with higher sample sizes. The study suggests that a five­ point response scale gives most reasonable results among various scales examined. IRT analysis is recommended as a preliminary process to ascertain the observed features of items. The study also finds a sample size of 150 as adequate for a most plausible factor solution, under various conditions.