Test Development and Validation

✍ Scribed by Gary Edward Skaggs

• Publisher: SAGE Publications, Inc
• Year: 2022
• Tongue: English
• Leaves: 409
• Edition: 1
• Category: Library

✦ Synopsis

Test Development and Validation by Gary Skaggs summarizes the latest test theories, frameworks for test development and validation, and guidance for developing tests in straightforward language in one core text. Students looking for clear, concise explanations of measurement, validity, and test development within a real-world context and with numerous examples will find this book to be an excellent learning resource. Author Gary Skaggs takes years of experience teaching test development to graduate students across social and behavioral sciences and consulting on a wide variety of government and institutional research projects to offer students a thorough, jargon-free, and highly applied book to help propel their own research and careers.

Part I of the book, The Big Picture, sets the stage for test development, placing it within the larger context and history of measurement, emphasizing measurement concepts and their evolution over time. Part II, Test Development, covers the technical details of instrument and test development in logical order. Validation, Part III, links the conceptual bases provided in Part I with the technical process provided in Part II to conclude the book. For those students wanting to go further, software suggestions are referenced in the technical chapters, while Further Reading sections offer the original sources for more details. Exercises and Activities at the end of each chapter provide students a variety of ways to apply their knowledge, from conceptual questions to brief project ideas to data analysis problems.

✦ Table of Contents

Cover
Title Page
Copyright Page
BRIEF CONTENTS
DETAILED CONTENTS
PREFACE
TEACHING RESOURCES
ACKNOWLEDGMENTS
ABOUT THE AUTHOR
Part 1. The Big Picture
Chapter
1. Measurement in the Social Sciences
1.1 What Do We Call Them?
1.2 What Is Measurement?
1.3 Some Key Terms
1.4 Measuring Constructs
1.5 Historical Context
1.5.1 Personnel Selection Testing in China, the United Kingdom, and United States
1.5.2 Credentialing
1.5.3 Measuring Intelligence
1.5.4 Educational Achievement
1.5.5 Personality
1.5.6 Current State of Test Development
1.6 Some Descriptive Statistics (Central Tendency, Variation, Correlation)
1.7 Sample Datasets: TIMSS and CIRP
1.8 Chapter Summary
1.9 Exercises and Activities
Further Reading
References
Chapter
2. The Evolution of Validity
2.1 First and Current Definitions of Validity
2.2 Criterion Model of Validity
2.3 Content-Based Validity Model
2.4 Construct Validity Model
2.5 The Holy Trinity
2.6 Messick's Validity Framework
2.6.1 Content Evidence
2.6.2 Substantive Evidence
2.6.3 Structural Evidence
2.6.4 External Evidence
2.6.5 Consequential Evidence
2.6.6 Generalizability
2.6.7 Integrating Validity Evidence
2.6.8 Standards and Messick's Framework
2.7 Kane's Argument-Based Validity Framework
2.7.1 Scoring Inference
2.7.2 Generalization Inference
2.7.3 Extrapolation Inference
2.7.4 Implication Inference
2.7.5 Theory-Based Interpretation Inference
2.7.6 Decision Inference
2.8 Current State of Validity and Validation
2.9 Chapter Summary
2.10 Exercises and Activities
Further Reading
References
Chapter
3. Test Scaling and Scoring
3.1 Scoring Items
3.2 Raw Scores for a Test
3.3 Handling Missing Data
3.4 Raw Scores Versus Scale Scores
3.5 Scaling for Norm-Referenced Score Interpretations
3.5.1 Percentiles and Percentile Ranks
3.5.2 Linear Transformations of Raw Scores
3.5.3 Nonlinear Transformations of Raw Scores
3.6 Scaling for Criterion-Referenced Score Interpretations
3.7 Equating/Linking Alternate Test Forms
3.7.1 Data Collection Designs for Equating
3.7.2 Three Classical Methods for Equating
3.8 Chapter Summary
3.9 Exercises and Activities
Further Reading
References
Chapter
4. Classical Test Theory
4.1 The Concept of Measurement Error
4.2 The Concept of Reliability
4.3 Theory of True and Error Scores
4.4 Reliability Coefficients
4.4.1 Reliability Coefficients Based on Parallel Measures
4.4.2 Reliability Coefficients Based on Other Models of Part-Test Similarity
4.4.3 Other Reliability Coefficients
4.4.4 What Is Good Reliability?
4.4.5 Reliability Coefficients for Real Data
4.5 Standard Error of Measurement
4.6 Relationship Between Reliability and Validity
4.7 Generalizability Theory
4.8 Chapter Summary
4.9 Exercises and Activities
Further Reading
References
A.1 Estimating the Reliability Coefficient Under Parallel Measures
A.2 Reliability of a Composite (Spearman–Brown Formula)
A.3 Standard Error of Measurement
Part 2. Test Development
Chapter
5. Frameworks for Test Development
5.1 Standards Framework for Test Development
5.2 Wilson's Framework for Test Construction
5.3 Evidence-Centered Design
5.4 Framework for This Book
5.5 Chapter Summary
5.6 Exercises and Activities
Further Reading
References
Chapter
6. Test Specifications
6.1 Defining the Purpose and Intended Uses
6.2 Several Early Decisions to Make
6.3 Creating a Construct Map
6.4 Internal Structure Model
6.5 External Relationship Model
6.6 Operationalizing the Construct
6.7 Test Specification Issues
6.8 Test Blueprints
6.8.1 Content Dimension
6.8.2 Process Dimension
6.9 Chapter Summary
6.10 Exercises and Activities
Further Reading
References
Chapter
7. Item Writing and Scoring
7.1 Items and Scoring Models
7.2 Item Formats Used Mostly for Cognitive Constructs
7.2.1 Multiple-Choice Items
7.2.2 Alternate Multiple-Choice Formats
7.2.3 Open-Ended Item Formats
7.2.3.1 Scoring Rubrics
7.2.3.2 Training Raters
7.2.3.3 Interrater Reliability
7.3 Item Formats Used Mostly for Noncognitive Constructs
7.3.1 Likert Items
7.3.2 Writing Likert Items
7.3.3 Likert Item Issues
7.3.4 Other Item Formats
7.4 Item Reviews
7.5 Chapter Summary
7.6 Exercises and Activities
Further Reading
References
Chapter
8. Pilot Testing and Item Analysis
8.1 Pilot Testing
8.1.1 Methods
8.1.2 Samples
8.1.3 Test Length
8.1.4 Design
8.1.5 Mode of Administration
8.2 Item Analysis
8.2.1 Item Analysis for Dichotomously Scored Items
8.2.1.1 Item Quality Indicators
8.2.1.2 Item Analysis of TIMSS Booklet 1
8.2.1.3 Selecting Multiple-Choice Items
8.2.2 Item Analysis for Polytomously Scored Items
8.2.2.1 Item Quality Indicators
8.2.2.2 Item Analysis of Political Viewpoint Data
8.2.2.3 Selecting Among Polytomously Scored Items
8.3 Chapter Summary
8.4 Exercises and Activities
Further Reading
References
Chapter
9. Introduction to Rasch Measurement Theory
9.1 Problems With Classical Test Theory
9.2 Development of Probability Models
9.3 Binary Rasch Model
9.4 Scaling and Scoring With the Rasch Model
9.5 Estimating Measurement Error With the Rasch Model
9.6 Assumptions and Model Fit
9.7 Example Using TIMSS Booklet 1 Data
9.8 Polytomous Rasch Models
9.8.1 The Partial Credit Model
9.8.2 Rating Scale Model
9.8.3 Example Using Political Viewpoint Data
9.9 Assumptions, Information, and LID
9.10 Comparing Rasch Ability Estimates With CTT Raw and Normalized z-Scores
9.11 Using Rasch Models in Test Development
9.12 Chapter Summary
9.13 Exercises and Activities
Further Reading
References
Part 3. Validation
Chapter
10. Fairness, DIF, and Item Bias
10.1 Fairness in Test Development and Use
10.2 Differential Item Functioning and Item Bias
10.3 Methods for Detecting Differential Item Functioning
10.4 DIF Methods for Dichotomously Scored Items
10.4.1 Mantel–Haenszel Methods
10.4.2 Mantel–Haenszel Analysis of TIMSS Items
10.4.3 SIBTEST
10.5 Mantel–Haenszel and SIBTEST Methods for Polytomously Scored Items
10.6 Other DIF Detection Methods
10.7 Using DIF Analysis as Validity Evidence
10.8 Chapter Summary
10.9 Exercises and Activities
Further Reading
References
Chapter
11. Dimensionality
11.1 Dimensionality, Internal Models, and Construct Maps
11.2 Methods for Assessing Dimensionality
11.3 Exploratory Factor Analysis
11.3.1 Extraction Methods
11.3.2 The Number of Factors
11.3.3 Rotation Methods
11.3.4 EFA Example Using the Political Viewpoint Data
11.3.5 EFA of Dichotomously Scored Data
11.4 Confirmatory Factor Analysis
11.5 Bifactor Model: An EFA That Looks Like a CFA
11.6 Using Factor Analysis Results as Validity Evidence
11.7 Chapter Summary
11.8 Exercises and Activities
Further Reading
References
Chapter
12. Standard Setting
12.1 Background
12.2 Standards for Standard Setting
12.3 The Process of Standard Setting
12.4 Standard Setting Methods
12.4.1 Borderline Group and Contrasting Group Methods
12.4.2 Angoff Methods
12.4.3 Bookmark Standard Setting Methods
12.4.4 Other Standard Setting Methods
12.5 Validating Standard Setting
12.6 Chapter Summary
12.7 Exercises and Activities
Further Reading
References
Chapter
13. Validity Evidence and Validation Plans
13.1 Current Validity Frameworks
13.2 Validation Using Messick/Standards Framework
13.3 Validation Plan for Grit Scale—Messick/Standards Version
13.4 Validation Using Kane's Framework
13.5 Validation Plan for Grit Scale—Kane Version
13.6 Chapter Summary
13.7 Book Summary
13.8 Exercises and Activities
References
Appendix A. Guide to Sample Data Sets
Appendix B. R Code for Analyses
Index