Educational Testing Service - Scientific Contributions

Scientific Contributions

In keeping with the purposes for which it was established, ETS developed a program of research that covered not only measurement and education but also such related areas as statistics, educational evaluation, and psychology, particularly cognitive, developmental, personality, and social psychology. This broad-based research program attracted many individuals who distinguished themselves in their fields, often while at ETS but also in subsequent professorial positions. Among the more influential scientists have been Harold Gulliksen (whose book, "Theory of Mental Tests," helped codify classical test theory); Frederic Lord (Item Response Theory); Samuel Messick, (modern validity theory); Robert Linn (currently known for testing and educational policy); Norman Frederiksen (performance assessment); Ledyard Tucker (test analysis, including inventing the "Angoff Method" of standard setting); Donald Rubin (missing data and causal modeling from observational data); Karl Joreskog (structural equation modeling and confirmatory factor analysis); Paul Holland (differential item functioning, test equating, causal modeling); John Carroll (language testing and cognitive psychology); Michael Lewis (infant cognitive, social, and emotional development); Irving Sigel (children's cognitive development); Herman Witkin (cognitive and learning styles); K. Patricia Cross (adult education); Samuel Ball (an evaluation researcher who documented the positive educational effects of Sesame Street); and David Rosenhan (known for the Rosenhan experiment, which challenged the validity of psychiatric diagnosis).

Members of the ETS staff have been among the presidents of the National Council on Measurement in Education (NCME); the Psychometric Society; the Measurement and Evaluation Division of the American Educational Research Association (AERA); the Evaluation, Measurement and Statistics Division of the American Psychological Association (APA); the APA Developmental Psychology Division; and the Jean Piaget Society. They have been among the executive editors of the Journal of Educational Measurement, Journal of Educational and Behavioral Statistics, Educational Evaluation and Policy Analysis, Journal of Educational Psychology, Journal of Applied Developmental Psychology, and Discourse Processes. Major citations have included the APA Distinguished Contributions to Knowledge Award (Norman Frederiksen, 1984), the APA Distinguished Scientific Contribution Award (Frederic Lord, 1988); the AERA E.F. Lindquist Award (William Turnbull, 1981; Frederic Lord, 1988; Samuel Messick, 1994; Paul Holland, 2000; Wendy Yen, 2008); the NCME Career Contributions to Educational Measurement Award (Frederic Lord, 1990; Paul Holland, 2004; Neil Dorans, 2010); and the Jean Piaget Society's Lifetime Achievement Award (Irving Sigel, 2002); among many other awards.

The high caliber of scientific staff allowed ETS to produce both new knowledge and methodology, especially in measurement and statistics, much of which has been taken up by assessment organizations around the world. Among the key scientific contributions were:

• co-invention of Item Response Theory, an integrated framework for asking and answering a variety of practical problems related to the design and analysis of tests;

• creation of an approach and software for structural equation modeling and confirmatory factor analysis (LISREL), used throughout the social sciences to test theoretical relationships among variables;

• seminal contributions to modern validity theory, including the idea that validity was a unitary concept and that the evaluation of score meaning requires consideration of the consequences of test use as those consequences may imply functional problems with the test;

• development of widely used approaches to data analysis when there are missing data;

• generation of approaches to causal modeling from observational data;

• invention of the In-Basket Test (used throughout the world to assess applicants for managerial jobs in a wide variety of industries);

• development of methods for detecting test unfairness, including invention of the Standardization approach to Differential Item Functioning (DIF) and application of the Mantel-Haenszel method;

• creation of the holistic-scoring approach to writing assessment, a means of rapidly and reliably judging the quality of essay text, which allowed direct writing assessment to become a more affordable alternative to multiple-choice questions for large-scale testing programs;