Computerized Adaptive Testing: From Concept to Implementation (Methodology in the Social Sciences Series)

✍ Scribed by David J. Weiss, Alper Sahin

• Publisher: The Guilford Press
• Year: 2024
• Tongue: English
• Leaves: 388
• Edition: 1
• Category: Library

✦ Synopsis

Used worldwide in assessment and professional certification contexts, computerized adaptive testing (CAT) offers a powerful means to measure individual differences or make classifications. This authoritative work from CAT pioneer David J. Weiss and Alper Şahin provides a complete how-to guide for planning and implementing an effective CAT to create a test unique to each person in real time. The book reviews the history of CAT and the basics of item response theory used in CAT. It walks the reader through developing an item bank, pretesting and linking items, selecting required CAT options, and using simulations to design a CAT. Available software for CAT delivery is described, including links to free and commercial options. Engaging multidisciplinary examples illustrate applications of CAT for measuring ability, achievement, proficiency, personality, attitudes, perceptions, patients’ reports of their symptoms, and academic or clinical progress.
 

✦ Table of Contents

Cover
Half Title Page
Series Page
Title Page
Copyright
Dedication
Series Editor’s Note
Preface
Contents
Part I. The Why, What, and How of CAT
1. Why CAT?
What Is CAT?
Benefits of CAT
Psychometric Benefits
Benefits for Examinees
Organizational Benefits
Challenges of CAT
Organizational Challenges
Examinee Challenges
Psychometric Challenges
When Is CAT Feasible and When Is It Not?
2. The Birth of Adaptive Testing and Its Current Status
Binet’s Adaptive Test
The Evolution of Adaptive Testing into CAT
Early Approaches to CAT
Two-Stage and Multistage Tests
CATs from “Pyramidal” Item Structures
The Self-Scoring Flexilevel Test
The Stratified-Adaptive Test
Deficiencies of the Early Approaches to CAT
The Current Status of Applications of CAT
3. Item Response Theory Used in CAT
Item Response Functions for Dichotomously Scored Items
Option Response Functions for Polytomously Scored Items
Information Functions
Item Information Functions
Test and Bank Information Functions
Measuring People with IRT
Maximum Likelihood Estimation of O
Variations of Maximum Likelihood Estimation
Linking and Equating
Estimating IRT Item Parameters
Selecting a Software Package
Interpreting the Output
Summary
Advanced Topic: Item Parameter Estimation for Complex Models
4. Using CAT to Measure Individual Differences
The Components of a Measurement CAT
The Item Bank
Beginning a CAT
Selecting the Next Item
Scoring Examinees
Ending the CAT
Putting It All Together: How CAT Works
Sample CAT: Examinee A
Sample CAT: Examinee B
Some Properties of Fully Adaptive IRT-Based CATs
Advanced Issues
CAT Item Selection
Ending a CAT
5. CAT for Classification and Classification Plus Measurement
Setting the Cutscore
The Bank Response Function
Types of Classification Applications
Testing for Classification Only
Adaptive Classification Test (ACT)
Testing for Classification Plus Measurement
Advanced Issues
Cutscore- versus Estimate-Based Item Selection
Ending an SCT When a O Estimate Is Too Close to the Cutscore
Classifying with More Than Two Cutscores
Classifying from Multiple Measurements
Part II. Getting Your CAT Operational
6. Feasibility and Planning
Developing a Business Case
The Item Bank
Options and Costs
Costs of Test Delivery
Delivering High-Stakes CATs
Custom Test Center Costs
Potential Risks
Psychometric Risks
Nonpsychometric Risks
Additional Considerations
Developing a Project Plan
Stakeholder Buy-In
Implementing Simulations to Facilitate CAT Design
Conclusions
7. Developing a CAT Item Bank: Writing and Evaluating Items
How Many Items Do I Need in My Bank?
Write as Many Items as You Can
Do Not Build an Item Bank Solely for Today—Think about the Future as Well
What Kinds of Items Work for CAT?
Developing High-Quality Items and Item Banks
Planning
Item Writing
Evaluating Your Items
Maintaining Your Item Bank
Characteristics of a Good Item Bank
Further Information
8. Pretesting and Linking Your Items, and Adding New Items to an Existing Bank
Pretesting Your Items
IRT Item (and Person) Parameter Linking
Conversion Methods of Linking
Full-Metric Item Parameter Linking
Seeding New Items into a CAT Item Bank
Additional Considerations in Item Bank Development
Advanced Topic: Automated Item Generation
9. Using Simulations to Design a CAT
Post-Hoc or Real-Data Simulation
Making a Conventional Test More Efficient
Designing a CAT from a CAT Item Bank
Monte Carlo Simulations
Hybrid Simulations
How Much Missing Data?
Conclusions
10. Implementing Your CAT
CAT Delivery Software
Free CAT Delivery Software
Commercial CAT Delivery Software
Build-Your-Own CAT Delivery System
How Can I Get CAT Operational for My Organization?
Stakeholder Buy-In
Other Resources
Part III. Applications, Special Types of CATs, Special Issues, and a Potential Future for CAT
11. Examples of Operational CAT Programs
CAT in Education
K–12 Formative/Interim/Summative Assessment
CAT for Military Accessioning
Armed Services Vocational Aptitude Battery (CAT-ASVAB)
Navy Computer Adaptive Personality Scales (NCAPS)
Higher Education Admissions and Placement
Computerized Adaptive Screening Test (CAST)
CAT for Pre-Employment Testing
Adjustable Competence Evaluation (ACE)
Computerized Adaptive Test for Ability (CATA)
CAT for Certification and Licensure
Certified Public Accountant (CPA) Exam
Emergency Medical Services (EMS) Exams
North American Pharmacist Licensure Examination (NAPLEX)
Multistate Pharmacy Jurisprudence Examination (MPJE)
National Council Licensure Examinations (NCLEX)
CAT in Healthcare
Mental Health
Patient-Reported Outcomes (PROs)
Additional Applications of CAT
12. Other Types of CATs
Multistage Tests
Two-Stage Tests
MSTs
Summary
CAT for Cognitive Diagnosis
Cognitive Diagnostic Assessment
Cognitive Diagnostic CAT (CD-CAT)
Summary
CAT for Measuring Multiple Variables
Multidimensional IRT
Multidimensional CAT (MD-CAT)
Summary
13. Special Issues
Constrained CATs
Content Balancing
Item Exposure
Multiple Constraints: Shadow Tests
CAT Using Items with Multiple Questions per Stimulus
CAT Administration Issues
Fixed-Length CATs
Time Limits
The Examinee Experience
A Suggested Preamble to a CAT
14. Adaptive Measurement of Change: Evaluating the Psychometric Significance of Measured Individual Change
Identifying Significant Individual Change Using an Early Adaptive Measurement of Change (AMC) Method
The AMC Procedure
The Likelihood Ratio Index (LRI)
Example AMC Applications
Univariate Change
Multivariate (Profile) Change
AMC Research
Discussion and Conclusions
Conclusions
15. Some Pathways toward Future CATs and CAT Research
Item Banking
AIG
On-the-Fly CAT
Chatbots as Item Writers
Test Delivery
Examinee Responses
More Realistic Test Items
Supplementing and Improving Scores
Response Times
Game-Based Tests
Other Process Data
Integration of Testing and Instruction
A Path for Future CAT Research: Does CAT Improve Validity?
A Sample Application of CAT SEMs in Predicting External Criteria
Method
Results
Summary and Conclusions
Leveraging Current and Future Technology to Improve Testing
Software Appendices
A. Software Packages for Estimating IRT Parameters
B. Software Packages for CAT Simulations
C. Software Platforms for CAT Delivery
Acronyms
References
Author Index
Subject Index
About the Authors
Symbols