e9781544339504_cover
FM-01-9781544339504
Outline placeholder
Praise for This Book
Brief Contents
Detailed Contents
Preface
Purpose
Audience
Book Structure
Book Features
Instructor Resources
Acknowledgments
About the Authors
PART-01-9781544339504
Part I. Fundamentals of Evaluation
CH001-9781544339504
1. The Scope of Evaluation
Introduction
What Is Evaluation?
Why Do Evaluation?
To Determine Merit, Worth, and Significance
To Gain an Understanding of How Programs Work and the Difference They Can Make to Stakeholders
To Improve Quality of Life
To Pursue Social Justice and a More Equitable World
Research and Evaluation
Evaluation and Research Are Mutually Exclusive
Evaluation Requires Research but Research Does Not Require Evaluation
Evaluation and Research Share Similarities
A Brief History of Evaluation
The Three Pillars of Evaluation
The Guiding Principles
The Program Evaluation Standards
The Professional Competencies
Education and Training
Professionalization
CH002-9781544339504
2. How Evaluators Think
Bev's Philosophy of Mornings
Introduction
Scaffolds for Evaluation Thought
The Ladder of Abstraction
The Ladder of Evaluation Theory
Theory in Action
Ontology: Our Worldview
Realism
Relativism
Paradigms or Frameworks
Positivism/Postpositivism
The Scientific Method
Plan-Do-Study-Act (PDSA) Cycle
Deductive Thinking
Constructivism
Inductive Thinking
Subjectivism
Reflective Thinking
Pragmatism
Abductive Thinking
Natural Science Theory
Systems Theory
The Socio-Ecological Model (SEM)
Chaos Theory
Complexity Theory
Social Science Theory
Knowledge
Attitude
Behavior
Evaluation Theory
The Program Evaluation Project
Epistemology: Our Knowledge
Importance of Coherence
Critical Thinking and Evaluative Thinking
CH003-9781544339504
3. Program Logic
The Golden Thread
Introduction
Theory of Change
Creating a Theory of Change
Project Superwomen Theory of Change
The Fiver Children's Foundation Theory of Change
Assessing Quality in Theories of Change
Program Theory
The IF-THEN Statement
Creating a Program Theory: Alberta Methadone Maintenance Guidelines Evaluation
Assessing Quality in a Program Theory
The Logic Model
The Purpose of a Logic Model
How to Read a Logic Model
The Structure of a Logic Model
Inputs
Activities
Outputs
Outcomes
Comparison of Logic Model Structures
Assessing the Quality of Logic Models
Other Logic Model Elements
Supporting Data Statement
Vision or Mission Statements
Goals of the Intervention
Assumptions
Contextual Factors
Reach
Advantages of Logic Models
Limitations of Logic Models
PART-02-9781544339504
Part II. Evaluation and the Program Life Cycle
CH004-9781544339504
4. Pre- and Early Program Evaluation
Grizzly Bears and Policy Change
Introduction
The Literature Review
Generic Literature Review
Systematic Review
Meta-Analysis
Stakeholder Analysis
Stakeholder Mapping
Social Network Analysis
Communicating with Stakeholders
Needs Assessment
Phases of a Needs Assessment
Phase 1. Pre-Assessment (exploration)
Phase 2. Assessment (data gathering)
Phase 3. Post-Assessment (utilization)
Mapping Assets and Needs
Social Indicators and Trend Analysis
Consensus Building
Brainstorming
Nominal Group Technique (NGT)
Delphi Method
Forums
Example of a Needs Assessment
Program Design
Nine Steps to Program Design
Step 1. Analyze Available Data
Step 2. Specify Problem Statement
Step 3. Identify an Intended Population
Step 4. Develop Program Description and Goal Statements
Step 5. Develop an Outcomes Measurement Framework
Step 6. Identify Appropriate Activities
Step 7. Build Effective Partnerships
Step 8. Provide a Realistic Timeline
Step 9. Identify Effective Evaluation Strategies
Evaluability Assessment
Evaluability Assessment Example: Engaging Stakeholders to Improve Data Collection in Evaluability Assessments
CH005-9781544339504
5. Mid-Cycle Program Evaluation
A Killer Asteroid Is Headed Our Way
Introduction
Formative Evaluation
Participatory Evaluation
Utilization-Focused Evaluation (U-FE)
Empowerment Evaluation
Developmental Evaluation
Process Evaluation
Coverage
Quality Improvement
Accreditation
Satisfaction and Feedback Loops
Conducting Effective Participatory Focus Group Research with
Systematically Marginalized Young People
Descriptive Evaluation
Document Review
Observation
Case Studies
Fidelity
Performance Management
Performance Indicators
Performance Standards
Performance Measurement
Monitoring and Evaluation (M & E)
CH006-9781544339504
6. End-of-Cycle Program Evaluation
Traffic Jam at 29,000 Feet
Introduction
Outcome Evaluation
Individual Change
Organizational Change
Community Change
Outcome Mapping and Outcome Harvesting
Impact Evaluation
Population Change
Changes in Health Indicators
Changes in Social Determinants of Health
The Counterfactual Argument
Contribution Analysis
Summative Evaluation
Determining Value and Worth
Sustainability Evaluation
Economic Evaluation
Cost-Effectiveness Analysis (CEA)
Cost-Benefit Analysis (CBA)
Cost-Utility Analysis (CUA)
Partial Economic Evaluations
Fiscal Analysis
PART-03-9781544339504
Part III. Evaluation Methods
CH007-9781544339504
7. Using Quantitative Methods in Evaluation
Data and the Dishwasher
Introduction
Overview of Quantitative Methods
What Is Credible Evidence?
Hierarchies and Tensions in Determining Credible Evidence
Rigor
Cultural Appropriateness
Types of Quantitative Data
Quantitative Designs
Terminology Used in Quantitative Design
Types of Quantitative Designs
Pre-Experimental Designs
Single Case Study
One-Group, Pretest–Posttest Design
Quasi-Experimental Designs
Nonequivalent Control Group Design
Interrupted Time Series
Experimental Designs
Posttest-Only Control Group Design
Pretest/Posttest Control Group Design
Strengths and Limitations of Quantitative Designs
Ensuring Quality in Quantitative Designs
Validity
Internal Validity
Threats to Internal Validity
External Validity
Threats to External Validity
Reliability
Internal Consistency
Test–Retest Reliability
Objectivity
Sensitivity
Analyzing Quantitative Data
Data Formats and Sources
Data Preparation
Hypothesis Testing
Statistical Analysis
Descriptive Statistics
Inferential Statistics
Parametric and Nonparametric Tests
The Dependent Sample t Test (or Paired Sample t Test)
Analysis of Variance
Pearson Correlation
Example of an Evaluation with a Quantitative Design: Factors Precipitating Suicidality Among Homeless Youth
The Survey: The Most Common Quantitative Tool
Is a Survey the Right Tool?
What Practical Considerations Are Needed When Designing a Survey?
How Can I Be Sure My Questions Will Yield the Data I Need?
CH008-9781544339504
8. Using Qualitative Methods in Evaluation
Tiny Babies and Qualitative Research
Introduction
Overview of Qualitative Methods
Quantitative Versus Qualitative—What's the Difference?
Ensuring Methodological Rigor in Qualitative Designs
What Is Qualitative Thinking?
Qualitative Designs
Qualitative Traditions
Ethnography
Phenomenology
Social Constructivism
Narrative Inquiry
Critical Theory
Grounded Theory
Common Characteristics of Qualitative Designs
Participants
Design
Data Collection and Analysis
The Self
Common Threats to Quality and How to Counteract Them
During Study Design
During Data Collection
During Data Analysis and Synthesis
Strengths and Limitations of Qualitative Designs
The Evaluator Experience in Old Crow, Yukon
Qualitative Analysis—Overview
Using Bloom's Taxonomy—A Framework for Qualitative Analysis
Levels 1 and 2—Knowledge and Comprehension
Prepare to Code
Level 3—Application
Record Analytic Thoughts and Processes
Level 4—Analysis
Reduce and Chunk the Data
Initial Coding
Categorizing
Theming
Level 5—Recombine and Interpret
Synthesis
Count
Note Patterns
Look at Plausibility
Cluster
Make Metaphors
Split Variables
Move From Particular to General
Factor
Note Relationships Among Variables
Find Intervening Variables
Data Displays
Narrative Description
The Matrix
The Flowchart
The Creative Graphic
Level 6—Making a Judgment
Evaluation
Building a Logical Chain of Evidence
Achieving Coherence
Example of an Evaluation With a Qualitative Design: The Girls Just Wanna Have Fun Evaluation
The Interview—The Most Common Qualitative Tool
Types of Interviews
Checklist for Planning the Interview
Checklist for Conducting the Interview
Trauma-informed Interviews
CH009-9781544339504
9. Using Mixed Methods in Evaluation
Jell-O® and Mixed Methods
Introduction
What Is Mixed Methods?
Advantages and Challenges of Mixed Methods Designs
Advantages of Mixed Methods Designs
Challenges of Mixed Methods Designs
Mixed Methods Thinking
Mixed Methods Designs
The Convergent Design
The Explanatory Sequential Design
The Exploratory Sequential Design
Mixed Methods Integration Strategies—A Rough Guide
Integrated Management Strategies
Integrated Design
Integrated Instrument Development
Integrated Data Analysis
Integrated Data Synthesis
Integrated Reporting
Integrating Through Narrative
Integrating Through Data Transformation
Integrating Through Joint Displays
Evaluation of An Alternate Lawyer Licensing Pathway Using Mixed Methods
Ensuring Quality in Mixed Methods Designs
Example of an Evaluation Using an Exploratory Sequential Mixed Methods Design: Telling It All—A Story of Women's Social Capital
The Focus Group: A Companion Tool in Mixed Methods Studies
Planning and Conducting the Focus Group
PART-04-9781544339504
Part IV. Communicating About Evaluation
CH010-9781544339504
10. The Evaluation Plan
The Franklin Expedition, A Planning Disaster
Introduction
The Preplanning Process: Case Study #1 The Learning Circle Pilot Project
Identify and Engage Stakeholders
Review Program Documents and Conduct a Brief Literature Review
Conduct Key Informant Interviews and On-Site Observation
Design the Program Theory and Logic Model
Ask the Evaluation Questions
Develop Indicators
Finalize Methods Decisions
Prepare the Data Collection Matrix
Creates a Shared Understanding of the Project
Acts as a Concrete Work Plan
Maintains Study Focus
Provides a Coding System for Study Tools
Guides Data Analysis
Provides a Systematic and Recognized Outline for the Final Report
The Evaluation Plan: Case Study #2 MD/PhD Educational Program
Title Page
Program Overview
Program Role
Program Description
Key Stakeholders
Program Context
Evaluation Description
Evaluation Purpose
Type of Evaluation
Evaluation Questions
Intended Users and Use
Evaluation Approach
Evaluation Standards, Ethics, and Values
Evaluation Methods
Data Analysis and Synthesis
Types of Analysis Planned
Methods of Data Synthesis
Role of Stakeholders in Data Interpretation
Reporting
Planned Evaluation Management: Work Samples
Team Roles
Management Strategy
Data Ownership and Security
Study Limitations and Strengths
Schedule of Activities
Budget
Accountability
CH011-9781544339504
11. Communication, Reporting, and Use
Emergency on Hallowe'en
Introduction
Communication During the Evaluation
Evaluation Conversations
Interactive Forms of Communication
Managing Conflict
Informal Communications Tools
Data Visualization
Visual Processing
Reporting at the End of the Evaluation
The Final Report
Classic Reporting
Action-Oriented Reporting
Report Impact
How to Craft Conclusions
Present Results in a Simple Format
Select Results That Are Dually Significant
Use Sensitivity Analysis
Test the Validity of the Conclusions
How to Craft Recommendations
Avoid Meek Recommendations
Be Politically Astute
Work Collaboratively With Stakeholders
Supporting Use Once the Evaluation Is Completed
Evaluation Use Within the Organization
Process Use
Organizational Learning
Evaluation Capacity Building
Evaluation Use Beyond the Organization
Knowledge Into Action
Knowledge Creation
Knowledge Dissemination
Knowledge Application
Evaluation Influence
Social Betterment
CH012-9781544339504
12. Evaluation Context and the Evaluator's New Stance
THE CREATION OF TURTLE ISLAND, AN OJIBWE LEGEND
Introduction
Context Matters
The Evaluator's Power and Responsibility
Zooming In
Program and Participants
Methods
Communication Strategies
Lessons From Zooming In
Zooming Out
The Global and Natural Imperative
Strategies for Evaluators
Use Systems Thinking
Seek Transformative Engagement
Incorporate Sustainability Into Evaluation Practice
Identity and Social Issues
Race
Strategies for Evaluators
Understand Racial Framing
Work Toward Cultural Humility
Look for Strengths, Not Deficits
Culture2
Strategies for the Evaluator
Integrate Indigenous Knowledge in Evaluation Practice
Work With Different Epistemologies
Use an Indigenous Evaluation Framework
Manage Indigenous Data With Respect
Ethnicity
Strategies for the Evaluator
Use the Socio-Ecological Model (SEM)
Consider LatCrit Strategies
Use Evaluation for Advocacy
Organizations and Structures
Strategies for the Evaluator
Expand Approach to Learning
Use Equitable Evaluation Strategies
The Evaluator's New Stance
BM-01-9781544339504
Glossary
BM-02-9781544339504
References
BM-03-9781544339504
Index