Cover
Half Title
Title Page
Copyright Page
Contents
Preface
Acknowledgments
About the Authors
Abbreviations
PART ONE. INTRODUCTION TO IMPACT EVALUATION
Chapter 1. Why Evaluate?
Evidence-Based Policy Making
What Is Impact Evaluation?
Prospective versus Retrospective Impact Evaluation
Efficacy Studies and Effectiveness Studies
Complementary Approaches
Ethical Considerations Regarding Impact Evaluation
Impact Evaluation for Policy Decisions
Deciding Whether to Carry Out an Impact Evaluation
Chapter 2. Preparing for an Evaluation
Initial Steps
Constructing a Theory of Change
Developing a Results Chain
Specifying Evaluation Questions
Selecting Outcome and Performance Indicators
Checklist: Getting Data for Your Indicators
PART TWO. HOW TO EVALUATE
Chapter 3. Causal Inference and Counterfactuals
Causal Inference
The Counterfactual
Two Counterfeit Estimates of the Counterfactual
Chapter 4. Randomized Assignment
Evaluating Programs Based on the Rules of Assignment
Randomized Assignment of Treatment
Checklist: Randomized Assignment
Chapter 5. Instrumental Variables
Evaluating Programs When Not Everyone Complies with Their Assignment
Types of Impact Estimates
Imperfect Compliance
Randomized Promotion as an Instrumental Variable
Checklist: Randomized Promotion as an Instrumental Variable
Chapter 6. Regression Discontinuity Design
Evaluating Programs That Use an Eligibility Index
Fuzzy Regression Discontinuity Design
Checking the Validity of the Regression Discontinuity Design
Limitations and Interpretation of the Regression Discontinuity Design Method
Checklist: Regression Discontinuity Design
Chapter 7. Difference-in-Differences
Evaluating a Program When the Rule of Assignment Is Less Clear
The Difference-in-Differences Method
How Is the Difference-in-Differences Method Helpful?
The “Equal Trends” Assumption in Difference-in-Differences
Limitations of the Difference-in-Differences Method
Checklist: Difference-in-Differences
Chapter 8. Matching
Constructing an Artificial Comparison Group
Propensity Score Matching
Combining Matching with Other Methods
Limitations of the Matching Method
Checklist: Matching
Chapter 9. Addressing Methodological Challenges
Heterogeneous Treatment Effects
Unintended Behavioral Effects
Imperfect Compliance
Spillovers
Attrition
Timing and Persistence of Effects
Chapter 10. Evaluating Multifaceted Programs
Evaluating Programs That Combine Several Treatment Options
Evaluating Programs with Varying Treatment Levels
Evaluating Multiple Interventions
PART THREE. HOW TO IMPLEMENT AN IMPACT EVALUATION
Chapter 11. Choosing an Impact Evaluation Method
Determining Which Method to Use for a Given Program
How a Program’s Rules of Operation Can Help Choose an Impact Evaluation Method
A Comparison of Impact Evaluation Methods
Finding the Smallest Feasible Unit of Intervention
Chapter 12. Managing an Impact Evaluation
Managing an Evaluation’s Team, Time, and Budget
Roles and Responsibilities of the Research and Policy Teams
Establishing Collaboration
How to Time the Evaluation
How to Budget for an Evaluation
Chapter 13. The Ethics and Science of Impact Evaluation
Managing Ethical and Credible Evaluations
The Ethics of Running Impact Evaluations
Ensuring Reliable and Credible Evaluations through Open Science
Checklist: An Ethical and Credible Impact Evaluation
Chapter 14. Disseminating Results and Achieving Policy Impact
A Solid Evidence Base for Policy
Tailoring a Communication Strategy to Different Audiences
Disseminating Results
PART FOUR. HOW TO GET DATA FOR AN IMPACT EVALUATION
Chapter 15. Choosing a Sample
Sampling and Power Calculations
Drawing a Sample
Deciding on the Size of a Sample for Impact Evaluation: Power Calculations
Chapter 16. Finding Adequate Sources of Data
Kinds of Data That Are Needed
Using Existing Quantitative Data
Collecting New Survey Data
Chapter 17. Conclusion
Impact Evaluations: Worthwhile but Complex Exercises
Checklist: Core Elements of a Well-Designed Impact Evaluation
Checklist: Tips to Mitigate Common Risks in Conducting an Impact Evaluation
Glossary
Boxes
1.1 How a Successful Evaluation Can Promote the Political Sustainability of a Development Program: Mexico’s Conditional Cash Transfer Program
1.2 The Policy Impact of an Innovative Preschool Model: Preschool and Early Childhood Development in Mozambique
1.3 Testing for the Generalizability of Results: A Multisite Evaluation of the “Graduation” Approach to Alleviate Extreme Poverty
1.4 Simulating Possible Project Effects through Structural Modeling: Building a Model to Test Alternative Designs Using Progresa Data in Mexico
1.5 A Mixed Method Evaluation in Action: Combining a Randomized Controlled Trial with an Ethnographic Study in India
1.6 Informing National Scale-Up through a Process Evaluation in Tanzania
1.7 Evaluating Cost-Effectiveness: Comparing Evaluations of Programs That Affect Learning in Primary Schools
1.8 Evaluating Innovative Programs: The Behavioural Insights Team in the United Kingdom
1.9 Evaluating Program Design Alternatives: Malnourishment and Cognitive Development in Colombia
1.10 The Impact Evaluation Cluster Approach: Strategically Building Evidence to Fill Knowledge Gaps
2.1 Articulating a Theory of Change: From Cement Floors to Happiness in Mexico
2.2 Mechanism Experiments
2.3 A High School Mathematics Reform: Formulating a Results Chains and Evaluation Question
3.1 The Counterfactual Problem: “Miss Unique” and the Cash Transfer Program
4.1 Randomized Assignment as a Valuable Operational Tool
4.2 Randomized Assignment as a Program Allocation Rule: Conditional Cash Transfers and Education in Mexico
4.3 Randomized Assignment of Grants to Improve Employment Prospects for Youth in Northern Uganda
4.4 Randomized Assignment of Water and Sanitation Interventions in Rural Bolivia
4.5 Randomized Assignment of Spring Water Protection to Improve Health in Kenya
4.6 Randomized Assignment of Information about HIV Risks to Curb Teen Pregnancy in Kenya
5.1 Using Instrumental Variables to Evaluate the Impact of Sesame Street on School Readiness
5.2 Using Instrumental Variables to Deal with Noncompliance in a School Voucher Program in Colombia
5.3 Randomized Promotion of Education Infrastructure Investments in Bolivia
6.1 Using Regression Discontinuity Design to Evaluate the Impact of Reducing School Fees on School Enrollment Rates in Colombia
6.2 Social Safety Nets Based on a Poverty Index in Jamaica
6.3 The Effect on School Performance of Grouping Students by Test Scores in Kenya
7.1 Using Difference-in-Differences to Understand the Impact of Electoral Incentives on School Dropout Rates in Brazil
7.2 Using Difference-in-Differences to Study the Effects of Police Deployment on Crime in Argentina
7.3 Testing the Assumption of Equal Trends: Water Privatization and Infant Mortality in Argentina
7.4 Testing the Assumption of Equal Trends: School Construction in Indonesia
8.1 Matched Difference-in-Differences: Rural Roads and Local Market Development in Vietnam
8.2 Matched Difference-in-Differences: Cement Floors, Child Health, and Maternal Happiness in Mexico
8.3 The Synthetic Control Method: The Economic Effects of a Terrorist Conflict in Spain
9.1 Folk Tales of Impact Evaluation: The Hawthorne Effect and the John Henry Effect
9.2 Negative Spillovers Due to General Equilibrium Effects: Job Placement Assistance and Labor Market Outcomes in France
9.3 Working with Spillovers: Deworming, Externalities, and Education in Kenya
9.4 Evaluating Spillover Effects: Conditional Cash Transfers and Spillovers in Mexico
9.5 Attrition in Studies with Long-Term Follow-Up: Early Childhood Development and Migration in Jamaica
9.6 Evaluating Long-Term Effects: Subsidies and Adoption of Insecticide-Treated Bed Nets in Kenya
10.1 Testing Program Intensity for Improving Adherence to Antiretroviral Treatment
10.2 Testing Program Alternatives for Monitoring Corruption in Indonesia
11.1 Cash Transfer Programs and the Minimum Level of Intervention
12.1 Guiding Principles for Engagement between the Policy and Evaluation Teams
12.2 General Outline of an Impact Evaluation Plan
12.3 Examples of Research–Policy Team Models
13.1 Trial Registries for the Social Sciences
14.1 The Policy Impact of an Innovative Preschool Model in Mozambique
14.2 Outreach and Dissemination Tools
14.3 Disseminating Impact Evaluations Effectively
14.4 Disseminating Impact Evaluations Online
14.5 Impact Evaluation Blogs
15.1 Random Sampling Is Not Sufficient for Impact Evaluation
16.1 Constructing a Data Set in the Evaluation of Argentina’s Plan Nacer
16.2 Using Census Data to Reevaluate the PRAF in Honduras
16.3 Designing and Formatting Questionnaires
16.4 Some Pros and Cons of Electronic Data Collection
16.5 Data Collection for the Evaluation of the Atención a Crisis Pilots in Nicaragua
16.6 Guidelines for Data Documentation and Storage
Figures
2.1 The Elements of a Results Chain
B2.2.1 Identifying a Mechanism Experiment from a Longer Results Chain
B2.3.1 A Results Chain for the High School Mathematics Curriculum Reform
2.2 The HISP Results Chain
3.1 The Perfect Clone
3.2 A Valid Comparison Group
3.3 Before-and-After Estimates of a Microfinance Program
4.1 Characteristics of Groups under Randomized Assignment of Treatment
4.2 Random Sampling and Randomized Assignment of Treatment
4.3 Steps in Randomized Assignment to Treatment
4.4 Using a Spreadsheet to Randomize Assignment to Treatment
4.5 Estimating Impact under Randomized Assignment
5.1 Randomized Assignment with Imperfect Compliance
5.2 Estimating the Local Average Treatment Effect under Randomized Assignment with Imperfect Compliance
5.3 Randomized Promotion
5.4 Estimating the Local Average Treatment Effect under Randomized Promotion
6.1 Rice Yield, Smaller Farms versus Larger Farms (Baseline)
6.2 Rice Yield, Smaller Farms versus Larger Farms (Follow-Up)
6.3 Compliance with Assignment
6.4 Manipulation of the Eligibility Index
6.5 HISP: Density of Households, by Baseline Poverty Index
6.6 Participation in HISP, by Baseline Poverty Index
6.7 Poverty Index and Health Expenditures, HISP, Two Years Later
7.1 The Difference-in-Differences Method
7.2 Difference-in-Differences When Outcome Trends Differ
8.1 Exact Matching on Four Characteristics
8.2 Propensity Score Matching and Common Support
8.3 Matching for HISP: Common Support
9.1 A Classic Example of Spillovers: Positive Externalities from Deworming School Children
10.1 Steps in Randomized Assignment of Two Levels of Treatment
10.2 Steps in Randomized Assignment of Two Interventions
10.3 Crossover Design for a Program with Two Interventions
15.1 Using a Sample to Infer Average Characteristics of the Population of Interest
15.2 A Valid Sampling Frame Covers the Entire Population of Interest
B15.1.1 Random Sampling among Noncomparable Groups of Participants and Nonparticipants
B15.1.2 Randomized Assignment of Program Benefits between a Treatment Group and a Comparison Group
15.3 A Large Sample Is More Likely to Resemble the Population of Interest
Tables
3.1 Evaluating HISP: Before-and-After Comparison
3.2 Evaluating HISP: Before-and-After with Regression Analysis
3.3 Evaluating HISP: Enrolled-Nonenrolled Comparison of Means
3.4 Evaluating HISP: Enrolled-Nonenrolled Regression Analysis
4.1 Evaluating HISP: Balance between Treatment and Comparison Villages at Baseline
4.2 Evaluating HISP: Randomized Assignment with Comparison of Means
4.3 Evaluating HISP: Randomized Assignment with Regression Analysis
5.1 Evaluating HISP: Randomized Promotion Comparison of Means
5.2 Evaluating HISP: Randomized Promotion with Regression Analysis
6.1 Evaluating HISP: Regression Discontinuity Design with Regression Analysis
7.1 Calculating the Difference-in-Differences (DD) Method
7.2 Evaluating HISP: Difference-in-Differences Comparison of Means
7.3 Evaluating HISP: Difference-in-Differences with Regression Analysis
8.1 Estimating the Propensity Score Based on Baseline Observed Characteristics
8.2 Evaluating HISP: Matching on Baseline Characteristics and Comparison of Means
8.3 Evaluating HISP: Matching on Baseline Characteristics and Regression Analysis
8.4 Evaluating HISP: Difference-in-Differences Combined with Matching on Baseline Characteristics
B10.1.1 Summary of Program Design
11.1 Relationship between a Program’s Operational Rules and Impact Evaluation Methods
11.2 Comparing Impact Evaluation Methods
12.1 Cost of Impact Evaluations of a Selection of World Bank–Supported Projects
12.2 Disaggregated Costs of a Selection of World Bank–Supported Impact Evaluations
12.3 Sample Budget for an Impact Evaluation
13.1 Ensuring Reliable and Credible Information for Policy through Open Science
14.1 Engaging Key Constituencies for Policy Impact: Why, When, and How
15.1 Examples of Clusters
15.2 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects, Power = 0.9
15.3 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects, Power = 0.8
15.4 Evaluating HISP+: Sample Size Required to Detect Various Minimum Desired Effects (Increase in Hospitalization Rate)
15.5 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects (Decrease in Household Health Expenditures)
15.6 Evaluating HISP+: Sample Size Required to Detect a US$2 Minimum Impact for Various Numbers of Clusters