Program Evaluation: Pragmatic Methods for Social Work and Human Service Agencies

✍ Scribed by Allen Rubin

• Publisher: Cambridge University Press
• Year: 2020
• Tongue: English
• Leaves: 508
• Category: Library

✦ Synopsis

Be prepared for your future role in a service-oriented agency. This textbook provides practical guidance on program evaluation while avoiding replicating other course material. Drawing on over 40 years of subject knowledge, Allen Rubin describes outcome designs that are feasible for service-oriented agencies and that match the degree of certainty needed by key users of outcome evaluations. The utility and easy calculation of within-group effect sizes are outlined, which enhance the value of evaluations that lack control groups. Instructions are also given on how to write and disseminate an evaluation report in a way maximizes its chances of being used. Conducting focus group interviews and capitalising on the value of non-probabilitysamples will become second nature after following the effective and pragmatic advice mapped out chapter-by-chapter.

✦ Table of Contents

Half title
Title page
Imprints page
Contents in Brief
Contents in Detail
Figures
Tables
Preface
Key Challenges for Program Evaluation Instructors and Students with Which This Book Aims to Help
Organization
Pedagogical Features
Supplemental Resources
Acknowledgments
Part I Introduction
Chapter 1 Introduction and Overview
Chapter Outline
1.1 Introduction
1.2 Why Evaluate?
1.3 Some Programs are Ineffective or Harmful
Critical Incident Stress Debriefing.
Scared Straight Programs.
1.4 Historical Overview of Program Evaluation
1.5 Evidence-Informed Practice
1.6 Philosophical Issues: What Makes Some Types of Evidence Better Than Other Types?
Contemporary Positivism.
Interpretivism.
Empowerment.
Constructivism.
1.7 Qualitative versus Quantitative Evaluations: A False Dichotomy
1.8 Definitions
1.9 Different Evaluation Purposes
1.10 Types of Evaluation
Summative Evaluation.
Formative Evaluation.
Process Evaluation.
Performance Measurement Systems.
Evaluating One’s Own Practice.
Accreditation.
1.11 Chapter Main Points
1.12 Exercises
1.13 Additional Reading
Chapter 2 Ethical and Cultural Issues in Program Evaluation
Chapter Outline
2.1 Introduction
2.2 Ethical Issues
2.3 Institutional Review Boards (IRBs)
2.4 Culturally Sensitive Program Evaluation
2.4.1 Recruitment
2.4.2 Retention
2.4.3 Data Collection
2.4.4 Analyzing and Interpreting Evaluation Findings
2.4.5 Measurement Equivalence
2.5 Developing Cultural Competence
2.5.1 Acculturation and Immigration
2.5.2 Subgroup Differences
2.5.3 Culturally Sensitive Data Analysis and Interpretation
2.6 Chapter Main Points
2.7 Exercises
2.8 Additional Reading
Part II Quantitative and Qualitative Methods for Formative and Process Evaluations
Chapter 3 Needs Assessment
Chapter Outline
3.1 Introduction
3.2 Defining Needs: Normative Need versus Felt Need
3.3 Felt Need versus Service Utilization
3.4 Needs Assessment Approaches
3.4.1 Social Indicators
Advantages/Disadvantages.
3.4.2 Rates under Treatment
Advantages/Disadvantages.
3.4.3 Key Informants
Advantages/Disadvantages.
3.4.4 Community Forums
Advantages/Disadvantages.
3.4.5 Focus Groups
How to Conduct a Focus Group.
Types and Sequence of Focus Group Questions.
Advantages/Disadvantages.
3.4.6 Community Surveys
Advantages/Disadvantages.
3.5 Triangulation
3.6 Chapter Main Points
3.7 Exercises
3.8 Additional Reading
Chapter 4 Survey Methods for Program Planning and Monitoring
Chapter Outline
4.1 Introduction
4.2 Samples, Populations, and Representativeness
Probability Sampling.
Non-probability Samples.
4.2.1 Non-response Bias
4.2.2 Sample Size
Maximizing Response Rates.
Follow-ups.
4.3 Recruiting Hard-to-Reach Populations
Tactics for Reaching and Recruiting Millennials.
4.4 Survey Modalities
4.5 Interviews
Be Prepared.
Professional Demeanor.
Be Punctual.
Starting the Interview.
Note Taking.
Use Neutral Probes.
4.6 Interview Guides
4.7 Client Satisfaction Surveys
Limitations.
4.8 Survey Questionnaire Construction
4.8.1 Guidelines for Item Wording
4.8.2 Guidelines for Questionnaire Format
4.9 Online Survey Questionnaire Preparation
4.10 Chapter Main Points
4.11 Exercises
4.12 Additional Reading
Part III Evaluating Outcome in Service-Oriented Agencies
Chapter 5 Selecting and Measuring Outcome Objectives
Chapter Outline
5.1 Introduction
5.2 Mission and Vision Statements
5.3 Logic Models
5.4 Stakeholder Goals
5.5 Triangulation
5.6 How to Write Good Program Outcome Objectives
5.7 Operationally Defining Objectives
5.7.1 Direct Observation
5.7.2 Self-Report
5.7.3 Available Records
5.8 How to Find and Select the Best Self-Report Outcome Measures
5.9 Criteria for Selecting a Self-Report Outcome Measure
5.9.1 Relevance
5.9.2 Feasibility
5.9.3 Reliability
5.9.4 Validity
5.9.5 Sensitivity
5.10 Chapter Main Points
5.11 Exercises
5.12 Additional Reading
Chapter 6 Inference and Logic in Pragmatic Outcome Evaluation: Don’t Let the Perfect Become the Enemy of the Good
Chapter Outline
6.1 Introduction
6.2 Causality Criteria Revisited
6.2.1 Correlation
6.2.2 Time Sequence
6.2.3 Ruling Out Alternative Explanations
6.3 Implications of Evidence-Informed Practice and Critical Thinking
6.4 A Caveat
6.5 A Successful Evaluator Is a Pragmatic Evaluator
6.6 Degree of Certainty Needed
6.7 Chapter Main Points
6.8 Exercises
6.9 Additional Reading
Chapter 7 Feasible Outcome Evaluation Designs
Chapter Outline
7.1 Introduction
7.2 Descriptive Outcome Evaluations
7.3 One-Group Pretest–Posttest Designs
7.4 Effect Sizes
7.4.1 Between-Group Effect Sizes
7.4.2 Within-Group Effect Sizes
7.5 Non-equivalent Comparison Groups Designs
7.6 Selectivity Biases
7.7 Switching Replication Design
7.8 Switching Replication Design Compared with Waitlist Quasi-experimental Design
7.9 Time-Series Designs
7.10 Choosing the Most Appropriate Design
7.11 Chapter Main Points
7.12 Exercises
7.13 Additional Reading
Chapter 8 Single-Case Designs for Evaluating Programs and Practice
Chapter Outline
8.1 Introduction
8.2 What Is a Single Case?
8.3 Overview of Single-Case Design Logic for Making Causal Inferences
Clinical Significance.
8.4 What to Measure and by Whom?
8.4.1 Obtrusive and Unobtrusive Observation
8.4.2 Quantification Options
8.5 Baselines
Are Baselines Ethical?
8.6 Alternative Single-Case Designs
8.6.1 The AB Design
8.6.2 The ABAB Design
8.6.3 The Multiple-Baseline Design
8.6.4 Multiple-Component Designs
8.7 B Designs to Evaluate the Implementation of Evidence-Supported Interventions
8.8 Using Single-Case Designs as Part of the Evidence-Informed Practice Process
8.9 Aggregating Single-Case Design Outcomes to Evaluate an Agency
8.10 Chapter Main Points
8.11 Exercises
8.12 Additional Reading
Chapter 9 Practical and Political Pitfalls in Outcome Evaluations
Chapter Outline
9.1 Introduction
9.2 Practical Pitfalls
9.2.1 Intervention Fidelity
9.2.2 Contamination of the Case Assignment Protocol
9.2.3 Recruiting Participants
9.2.4 Retaining Participants
9.3 Engage Agency Staff Meaningfully in Planning the Evaluation
9.4 Fostering Staff Compliance with the Evaluation Protocol Goes On and On
9.5 Political Pitfalls
9.5.1 In-House versus External Evaluators
9.6 Conclusion
9.7 Chapter Main Points
9.8 Exercises
9.9 Additional Reading
Part IV Analyzing and Presenting Data
Chapter 10 Analyzing and Presenting Data from Formative and Process Evaluations
Chapter Outline
10.1 Introduction
10.2 Quantitative and Qualitative Data Analyses: Distinctions and Compatibility
10.3 Descriptive Statistics
10.3.1 Frequency Distribution Tables and Charts
10.3.2 Central Tendency
10.3.3 Dispersion
10.3.4 The Influence of Outliers
10.4 Analyzing Qualitative Data
10.4.1 Coding
10.5 Chapter Main Points
10.6 Exercises
10.7 Additional Reading
Chapter 11 Analyzing Data from Outcome Evaluations
Chapter Outline
11.1 Introduction
11.2 Inferential Statistics
11.2.1 p Values and Significance Levels
11.2.2 Type II Errors
11.3 Mistakes to Avoid When Interpreting Inferential Statistics
11.3.1 Overreliance on Statistical Significance
11.3.2 Disregarding Sample Size (Statistical Power)
11.3.3 Disregarding Effect Sizes
11.4 Calculating and Interpreting Effect Sizes
11.4.1 Within-Group Effect Sizes
Step 1.
Step 2.
Step 3.
Step 4.
Step 5.
11.4.2 Between-Group Effect Sizes
11.4.3 Why Divide by the Standard Deviation?
11.4.4 A Caution
11.4.5 Odds Ratios and Risk Ratios
Odds Ratios.
Risk Ratios.
11.5 Overlooking Substantive (Practical) Significance
11.6 Cost-Effectiveness and Cost–Benefit Analyses: Evaluating Efficiency
11.7 Qualitative Data Analysis
11.8 Chapter Main Points
11.9 Exercises
11.10 Additional Reading
Chapter 12 Writing and Disseminating Evaluation Reports
Chapter Outline
12.1 Introduction
12.2 Tailor to Your Audience
12.3 Writing Style and Format
12.4 Involve Key Stakeholders
12.5 Ethical Issues
12.6 Report Components
12.6.1 Executive Summary
12.6.2 Introduction and Literature Review
12.6.3 Methodology
12.6.4 Results (Findings)
Infographics.
12.6.5 Discussion
Discussing Negative Findings.
What If Parts of the Evaluation Could Not Be Completed?
12.6.6 References
12.6.7 Appendices
12.7 Summary of Mistakes to Avoid
12.8 Dissemination
12.9 Chapter Main Points
12.10 Exercises
12.11 Additional Reading
Epilogue More Tips for Becoming a Successful Evaluator
Planning the Evaluation
Levels of Stakeholder Participation.
Obtain Feedback to a Written Draft of the Evaluation Protocol.
During Implementation of the Evaluation
At the Conclusion of the Evaluation
People Skills
Show Genuine Interest in Others.
Try to Be Humorous.
Be Self-Assured.
Show Genuine Empathy.
Active Listening.
References
Index