The Conjoint Additivity of the Lexile Framework for Reading

Andrew Kyngdon, PhD
akyngdon@lexile.com
MetaMetrics, Inc

Reading is one of the most important skills a person acquires. The Lexile Framework for Reading is a system for the measurement of reading which matches continuous prose text to reader ability. It argues that comprehension of continuous prose text is a non-interactive, additive function of reader ability and text difficulty, consistent with measurement as argued by the Rasch (1960) model. However, the Framework has been publicly criticized by Joseph Martineau (2006) as not providing interval scale measurement of reader ability. In part to address this criticism, the Framework was tested with the theory of conjoint measurement (Luce & Tukey, 1964). If the cancellation axioms of this theory are supported, then it is a mathematically proven consequence that the relevant attributes are measurable at the interval scale level. Given the ordinal constraints imposed by these axioms, an order restricted inference approach must be taken in probabilistically testing them with noisy data (Iverson & Falmagne, 1985). Karabatsos' (2001) Bayesian Markov Chain Monte Carlo methodology was thus used to analyze data taken from 130 North Carolina 2nd grade children responding to 200 native Lexile text passage items. It was found both the single and double cancellation axioms of conjoint measurement were probabilistically satisfied. Such results support the conceptual argument of the Lexile Framework and that it provides measures of reader ability and text difficulty on the same interval scale. Martineau's (2006) criticisms were thus found to have no base.

Perspective / theoretical framework

Reading is one of the most important skills a person acquires. The Lexile Framework for Reading is a system for the measurement of reading which matches continuous prose text to reader ability. It argues that the difficulty of continuous prose text is a functional composition of two key cognitive variables - syntactic complexity and semantic rarity. Each is assessed using a proxy variable. For syntactic complexity, it is the natural logarithm of the mean sentence length (LMSL), which is a good proxy for the demand prose text places upon working memory (e.g. Klare, 1963; Crain & Shankweiler, 1988). For semantic rarity, the proxy variable is the mean natural logarithm of word frequency (MLWF). Stenner, Smith & Burdick, (1983) found MLWF was the best predictor of text difficulty out of 50 tested, including part of speech, modal grade, word content and number of syllables.

The difficulty of a prose text passage i, Di, is a composition of these proxies such that: Di = (a x LMSL) - (b x MLFW) - c. This is referred to as a construct specification equation, where a, b and c are real valued constants. The actual values of these constants are proprietary; however, they result in Di being a scale in logit units.
The Lexile theory argues that the probability of comprehending text is a non-interactive, additive function of the text's difficulty and the persons' reading ability. A suitable model for this is a modified Rasch (1960) model where Pr [Person v comprehends text passage i] = exp(Bv - Di + k) / 1 + exp(Bv - Di + k), where Bv is the ability of person v and k is a constant which makes the response probability equal to .75 when person ability and item difficulty coincide rather than the standard .50.

Objectives

The biggest weakness of the Lexile theory is the assumption of something more than monotone relationships existing between proxy variables and the relevant psychological attributes (Krantz & Tversky, 1971). Moreover, the Lexile theory proposes that text comprehension is a conjoint system, but this has not been tested independently of Rasch (1960) model analyses.

Joseph Martineau (2006) publicly criticised the Lexile Framework, arguing there was little reason to suggest that the Framework provided interval scale measurement. He argued, without any supporting evidence, that "Corporations with a monetary interest tend to make the claims below [of interval scale measurement] often and with impunity". He added that profit was a strong motive for "...claiming properties of linearity, unidimensionality and interval level measurement". He concluded that "Adequate psychometric models do not exist to substantiate these claims".

In actual fact, all of the above concerns can be tested by applying the theory of conjoint measurement (Luce & Tukey, 1964; Krantz, Luce, Suppes & Tversky, 1971) to the Lexile Framework for Reading. If the axioms of this theory hold, then it is a mathematically proven consequence that the relevant attributes can be measured at the interval scale level.

Methods / Techniques

Software and data analysis model

Given the ordinal constraints imposed by the cancellation axioms, an order restricted inference approach must be taken in probabilistically testing them with noisy data (Iverson & Falmagne, 1985). Karabatsos' (2001) Bayesian Markov Chain Monte Carlo methodology was thus used. The MCMC algorithm characterises a hybrid Metropolis Hastings Gibbs sampler suitable for order restricted inference. Karabatsos' (2001) S - Plus program was modified by the author to run in the "R" software package (R Development Core Team, 2007). A computational change was made from Karabatsos' original program in that the default method used to calculate the posterior quantiles was changed to the unbiased method recommended by Hyndman & Fan (1996).

Procedure

As 200 items were included in the present study there was a total number of 2^200 = 1.61 x 1060 possible test score patterns. With a sample of 130 people this meant it was highly unlikely that any two individuals would obtain the same test score pattern at random; and therefore unlikely that very many persons would obtain the same total score. This was borne out by the data. Only four subjects at most obtained the same total score. As a result it was decided not to form row vectors for a two way conjoint table on the basis of individuals having the same total score. Instead, 5 row vectors were formed by evenly dividing the sample size into 5 groups of 26 persons based on a total score range.

Also with 200 items produces an unmanageably large number of tests of the double cancellation axiom. Thus the theoretical Lexile measure of each item (obtained from the construct specification equation) was used to order the items from "easiest" to "hardest". The items were then divided in 4 groups. A 5 x 4 conjoint table was thus produced in which the cells comprised of 1300 person - item interactions (26 people to 50 items).

Enumeration studies have enabled researchers to calculate the probability of two way tables supporting the conjoint measurement axioms at random. Using the tables and formulae provided by McClelland (1977) & Ullrich & Wilson (1993), the probability of a 5 x 4 two way table satisfying single cancellation at random is approximately 1.968 x 10-9. It is thus quite unlikely that single cancellation will be satisfied at random by a table of this size; and so hence a very stringent test of this axiom is obtained.

Data Source

One hundred and thirty 2nd grade children from an elementary school in Parkwood, North Carolina, USA responded to 200 dichotomous text passage items.

Results

Each cell of 5 x 4 table consisted of the observed proportion of correct person - item encounters out of a total of 1300 (26 people, 50 items). The MCMC methodology probabilistically tests the single cancellation axiom of conjoint measurement by calculating posterior 95% credibility intervals calculated from the quantiles of the posterior distributions. All proportions were contained within the MCMC 95% posterior credibility intervals and as such the single cancellation axiom was probabilistically supported.

Forty instances of double cancellation are obtained from a 5 x 4 table and must be tested. The probability of a 5 x 4 table satisfying double cancellation at random given single cancellation holds is .122 (McClelland, 1977; Ullrich & Wilson, 1993).

Each instance was subject to test using the Bayesian MCMC model used to test the single cancellation axiom. All 40 instances of double cancellation were supported by the MCMC analysis, as all proportions were located within their respective 95% posterior credibility intervals.

Conclusions

The Lexile theory argues that the probability of comprehending a written prose text passage is a non-interactive, additive function of the text's difficulty and the persons' reading ability. If true, this theory is an instance of additive conjoint measurement. The results of the present study support this conclusion. The order restricted Bayesian MCMC inference model (Karabatsos, 2001) found that the cancellation axioms of the theory of conjoint measurement (Luce & Tukey, 1964) were probabilistically supported. This is evidence consistent with the hypothesis advanced by the Lexile system that reader ability and the difficulty of continuous prose text are quantitative attributes measurable on the same interval scale. Martineau's (2006) criticisms were thus found to have no base.

References

Crain, S. & Shankweiler, D. (1988). Syntactic complexity and reading acquisition. In A. Davidson and G.M. Green (Eds.), Linguistic complexity and text comprehension: readability issues reconsidered. Hillsdale, N.J.: Erlbaum.

Hyndman, R. J. and Fan, Y. (1996) Sample quantiles in statistical packages. American Statistician, 50, 361-365.

Iverson, G. & Falmagne, J.C. (1985). Statistical issues in measurement. Mathematical Social Sciences, 10, 131-153.

Karabatsos, G. (2001). The Rasch model, additive conjoint measurement, and new models of probabilistic measurement theory. Journal of Applied Measurement, 2, 389 - 423.

Klare, G.R. (1963) The measurement of readability. Ames, IA: Iowa State University Press.

Krantz, D.H.; Luce, R.D; Suppes, P. & Tversky, A. (1971). Foundations of Measurement, Vol. I: Additive and polynomial representations. New York: Academic Press.

Krantz, D.H. & Tversky, A. (1971). Conjoint measurement analysis of composition rules in psychology. Psychological Review, 78, 151-169.

Luce, R.D. & Tukey, J.W. (1964). Simultaneous conjoint measurement: a new scale type of fundamental measurement. Journal of Mathematical Psychology, 1, 1-27.

Martineau, J. (2006). Non-linear unidimensional scale trajectories through multidimensional content spaces: a critical examination of the common psychometric claims of unidimensionality, linearity and interval level measurement. Paper presented at the "Assessing and Modelling Cognitive Development at School: Intellectual Growth and Standard Setting" conference, Maryland Assessment Research Centre for Educational Success.

McClelland, G. (1977). A note on Arbuckle and Larimer. The number of two way tables satisfying certain additivity axioms. Journal of Mathematical Psychology, 15, 292-295.

Michell, J. (1988). Some problems in testing the double cancellation condition in conjoint measurement. Journal of Mathematical Psychology, 32, 466-473.

R Development Core Team. (2007). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.

Rasch, G. (1960).  Probabilistic models for some intelligence and attainment tests.  Copenhagen: Danish Institute for Educational Research.

Stenner, A.J., Smith, M. & Burdick, D. S. (1983) Toward a theory of construct definition. Journal of Education Measurement, 20, 305-315.

Ullrich, J.R. & Wilson, R.E. (1993). A note on the exact number of two and three way tables satisfying conjoint measurement and additivity axioms. Journal of Mathematical Psychology, 37, 624-628.