Evaluation in Today’s World

✍ Scribed by Thomas, Veronica G. & Campbell, Patricia B

• Publisher: SAGE Publications, Inc.
• Year: 2021
• Tongue: English
• Leaves: 1537
• Category: Library

✦ Table of Contents

Preface
About the Authors
Acknowledgments
Chapter 1 Evaluations of Future: Inclusive, Equity-Focused, Useful, and Used
Introduction
An Overview of the Book
Structure of the Book
Chapter Content
An Overview of Evaluation
Definitions of Evaluation
Evaluation Characteristics
Evaluative Thinking
Race, Racism, Social Justice, and a Racialized Perspective
Other Social Justice Issues
Objectivity and Bias
Objectivity
Bias
Explicit Bias
Implicit Bias
Reducing Bias
Culture, Cultural Competence, and Cultural Responsiveness
The Impact of Politics
The Current Climate
Summary
Supplemental Resources
Chapter 2 Evaluation Ethics and Quality Standards
Introduction
A Brief Historical Perspective on Research Ethics
The Nuremberg Code of 1947
The Tuskegee Syphilis Study of 1932–1972
The Radiation Studies of 1940–1960
The HeLa Story: 1950s and Beyond
Beyond Medical Studies and Physical Harm: The Milgram Study of 1963
The National Research Act of 1974
The Continuing Importance of Research Ethics
Ethics in Evaluation
Sources of Ethical Thinking
Cultural Competence as an Ethical Imperative
Ethical Dimension of Racial Bias
Ethical Sensitivity and Dilemmas
Sources of Ethical Dilemmas
Handling Ethical Dilemmas
Ethics and Conflicts of Interest
Ethical Challenges and Dilemmas Across the Evaluation Process
Ethical Principles and Standards for Evaluators and Evaluations
The Evaluators’ Ethical Guiding Principles
The Program Evaluation Standards
Evaluation Corruptibility and Fallacies
Evaluator Role, Power, Politics, and Ethics
Interplay of Politics and Ethics
Summary
Supplemental Resources
Chapter 3 Historical Evolution of Program Evaluation Through a Social Justice Lens
Introduction
History of Evaluation Through a Social Justice Lens
Evaluation Prior to Modern Times of the 20th Century
Intersection Between Education and Evaluation Pre–20th Century
Early Social Experiments
Overview of Evaluation in the 20th Century
Evaluation in the First Half of the 20th Century: 1900–1950s
Evaluation During the New Deal, Wartime, and Economic Growth: 1930s–1950s
The Cambridge-Somerville Youth Program Evaluation
Sputnik’s Impact on the Growth of Evaluation
Prominent Influencers and Users of Evaluation During the 20th-Century Early Years: 1930s–1950s
Kurt Lewin
Alva and Gunnar Myrdal
Ralph W. Tyler
Hidden Figures and Histories in Early-20th-Century Evaluation
Ambrose Caliver
Reid E. Jackson
Rose Butler Browne
Aaron A. Brown
Leander L. Boykin
Journal of Negro Education and Founding Editor Charles H. Thompson
Evaluation in 1960–2000
Federal Legislation and Great Society Programs
The Professionalization of the Field
Growth of Evaluation Scholarship
Establishment of Professional Societies in Evaluation
Graduate Training and Professional Development in Evaluation
Establishment of Standards and Codes of Conduct
Methodological Approaches and Paradigm Wars
Two Influential Scholars’ Contributions to Methodological Approaches of the 1960s–1970s
Rethinking the Role of Evaluation
Influential Women in Evaluation: 1970s–1990s
Carol H. Weiss
Yvonna S. Lincoln
Eleanor Chelimsky
Floraline I. Stevens
Lois-Ellin Datta
Laura Leviton
Beatriz Chu Clewell
Influential 20th-Century Evaluator: An Activity
21st-Century Evaluation: Expanding the Focus
Strengthening Evaluation at the Federal Level
Shift in the Quantitative–Qualitative Debate
Increased Emphasis on Social Justice and Diversity
Support for Capacity Building
Summary
Supplemental Resources
Chapter 4 Evaluation Paradigms, Theories, and Models
Introduction
The Value of Scientific Paradigms and Theories in Evaluation
The Nature of Scientific Paradigms
Theories for Guiding and Improving Evaluation Practice
Social Science Paradigms and Theories
Social Science Paradigms
Application of Social Science Paradigms in Evaluations
Social Science Theories
Program Theory of Change
Evaluating Program Theory
Evaluation Theories, Models, and Approaches
Why Should We Care About Evaluation Theory?
Distinguishing Evaluation Theories, Models, and Approaches
Classifying Evaluation Approaches and Theories
Five-Category Classification
The Evaluation Tree
Mertens and Wilson’s Four-Branch Tree of Evaluation Approaches
Evaluation Theories Within a Cultural Context
Evaluation Approaches and Theories: A Summary Description of Selected Examples
Summary
Supplemental Resources
Chapter 5 Social Justice and Evaluation: Theories, Challenges, Frameworks, and Paradigms
Introduction
Social Justice
Definitions of Social Justice
Marginalized Groups
Impacts of Marginalization
Fixing the Group vs. Fixing the System
Theories Providing Context for Social Justice Evaluations
Critical Race Theory
Feminist Research and Theory
Queer Theory
Disability Theory
Race, Racism, and Evaluation
Challenges to Social Justice and Evaluation
Traditional Definitions of Rigor
Deficit Models
Cultural Conflict of Interest
Efforts to Reduce the Impact of Racism on Evaluation
Cultural Competence and Cultural Responsiveness
Evaluation Models and Social Justice
Social Justice–Oriented Evaluation Frameworks and Paradigms
Transformational Evaluation
Empowerment Evaluation
Feminist Evaluation
Participatory Evaluation
Deliberative Democratic Evaluation
Collaborative Evaluation
Equity-Focused Evaluation
Summary
Supplemental Resources
Chapter 6 Evaluation Types With a Cultural and Racial Equity Lens
Introduction
Classifying Evaluations
An Overview of Formative and Summative Classification
Distinguishing and Coupling Formative and Summative Evaluations
Other Evaluation Classifications
Different Types of Evaluations
Formative and Implementation Evaluations
Needs Assessments
Evaluability Assessments
Process Evaluations
Progress Evaluations
Summative, Outcome, and Impact Evaluation Types
Outcome Evaluations
Impact Evaluations
Efficiency Evaluations
Alternative Types of Evaluations
Rapid Evaluations
Metaevaluations
Developmental Evaluations: Another Alternative to Formative–Summative
Putting It All Together
Summary
Supplemental Resources
Chapter 7 Social Programming, Social Justice, and Evaluation
Introduction
Understanding Social Problems and Social Programs Through a Social Justice Lens
Wicked Problems
Social Problems: Definition, Description, and Theoretical Underpinnings
Objective Element of Social Problems
Subjective Element of Social Problems
Sources of Social Problems
Social Problems’ Fluid Nature
Social Programs Through a Social Justice and Transformative Lens
Power, Political, and Economic Nature of Social Problems
Equity-Based Social Programming
Structural Racism, Social Programming, and Evaluation
Integrating Program Planning and Evaluation Planning
Social Program Evaluations vs. Social Project Evaluations: Distinctions and Implications
Key Program/Project Components Every Evaluator Must Understand
Program Mission
Program Goals
Program Objectives
Program Activities
Program Resources
Putting It All Together: Program Mission, Goals, Objectives, Activities, and Resources
Logic Models: Linking Program Components
Benefits of Logic Models for Program Planning
Limitations of Logic Models
Logic Models and Evaluation Planning
Components of Logic Models
Types and Looks of Logic Models
Nested Logic Models
Beyond Traditional Linear Logic Models
Fuzzy Logic Models
Circular Logic Models
Culturally Relevant Logic Models
Afrocentric-Centered Logic Model Approaches
Logic Models From an Indigenous Framework
Summary
Supplemental Resources
Chapter 8 Responsive Stakeholder Engagement and Democratization of the Evaluation Process
Introduction
Who Are Stakeholders?
Valuing Stakeholders and Diverse Stakeholder Engagement
Identifying and Classifying the Right Stakeholders
Stakeholder Classifications
Key and Hidden Stakeholders
Democratizing the Evaluation Process With Stakeholders
Relationships, Values, and Stakeholder Engagement
Responsive Stakeholder Engagement
The Misuse of Responsive Stakeholder Engagement
Continuum of Stakeholder Engagement: From Nonresponsive to Responsive
Barriers to Responsive Stakeholder Engagement
Benefits of Responsive Stakeholder Engagement
Six-Step Process for Responsive Stakeholder Engagement
Communicating With Stakeholders
Summary
Supplemental Resources
Chapter 9 Planning the Evaluation
Introduction
Dealing With Power Imbalances During Evaluation Planning
Planning for Culturally Responsive and Social Justice–Oriented Evaluations
Evaluation Planning Activities
Identifying and Involving Stakeholders in Evaluation Planning
Identifying Stakeholders and Their Potential Role(s)
Involving Stakeholders
Analysis of the Context
Identifying and Clarifying Project Goals
Identifying the Purpose(s) of the Evaluation
Varying Stakeholders’ Perspectives on Evaluation Goals and Priorities
Funder’s Priorities
Project Administrators’ Priorities
Evaluator’s Priorities
Defining Success in Evaluation Planning
The Problem With “Parity” as Success Definition
Beyond Quantitative Definitions of Success
Identifying Indicators
Understanding Different Types of Indicators
Levels of Indicators
Developing Timelines
Identifying Resource Needs
Assembling an Evaluation Team
Identifying the Evaluation Team’s Roles and Responsibilities
Internal vs. External Evaluators
Evaluation Planning and Management Visualization Tools
Gantt Charts
Program Evaluation Review Technique (PERT) Charts
Time and Task Charts or Data Maps
Developing a Written Evaluation Plan
Overcoming Pitfalls in Evaluation Planning
Summary
Supplemental Resources
Chapter 10 Evaluation Questions That Matter
Introduction
Why Evaluation Questions That Matter?
Questions That Matter Meet Information Needs of Diverse Users
Questions That Matter Set the Stage for the Collection of Credible Evidence
Power and Privilege Issues in Formulating Evaluation Questions
Characteristics of Good Evaluation Questions: An Overview
Good Questions Align With the Funder’s Requirements
Good Questions Are Useful and Ask About Important Issues
Good Questions Are Tailored and Appropriate to Local Needs
Good Questions Are Clear, Specific, and Well Defined
Good Questions Are Researchable (or Answerable)
Good Questions Are Realistic Considering Contexts and Project Realities
Good Questions Are Reasonable in Number and Scope
Sources of Evaluation Questions
Prioritizing Evaluation Questions for Diverse Audiences
Inclusion/Exclusion Criteria for Prioritizing Evaluation Questions: Two Approaches
U.S. Agency for International Development (USAID) Approach
Centers for Disease Control and Prevention (CDC) Approach
Steps to Identifying, Formulating, and Prioritizing Questions That Matter
Types of Evaluation Questions
Program Theory Questions
Context Questions
Process Questions
Relevance Questions
Outcomes Questions
Impact Questions
Cost-Benefit and Cost-Effectiveness Questions
Sustainability Questions
Summary of Different Types of Evaluation Questions
Summary
Supplemental Resources
Chapter 11 Selecting Appropriate Evaluation Designs
Introduction
Rigor
Bias
Practical Considerations
Theoretical and Cultural Considerations
Control and Comparison Groups
Ethical Issues
Ethical Use of Control and Comparison Groups
Longitudinal Data
Evaluation Designs
Experimental Designs
Feasibility of Implementation
Quasi-experimental Designs
Feasibility of Implementation
Pretest/Posttest Designs
Feasibility of Implementation
Retrospective Pretest Designs
Feasibility of Implementation
Case Studies/Ethnography
Feasibility of Implementation
Rival Hypotheses and Threats to Validity
The Best Design for the Question
Summary
Supplementary Resources
Chapter 12 Defining, Collecting, and Managing Data
Introduction
Qualitative and Quantitative Data
Sources of Qualitative Data
Interviews
Focus Groups
Observations
Participant-Generated Visual Data
Sources of Quantitative Data
Surveys and Other Structured Questionnaires
Records and Other Archival Data
Strengths and Weaknesses of Qualitative and Quantitative Data
Ensuring Data Quality
Validity
Types of Validity
Reliability
Reliability of Quantitative Data
Reliability of Qualitative Data
Pilot-Testing
Response Rates
Protection of Human Participants
Using Existing Measures or Developing New Ones
Sources of Measures
Assessing Existing Measures
Measuring Complex Concepts
Modes of Data Collection
Data Management
Mapping Data Collection to Project Goals and Objectives
Timing
Electronic Controls and Data Cleaning
Privacy
Data Management Plans
Summary
Supplemental Resources
Chapter 13 The Best Analysis for the Data
Introduction
Deductive and Inductive Reasoning
An Overview of Deductive and Inductive Reasoning
Using Deductive and Inductive Reasoning
Quantitative Analysis
Levels of Quantitative Data
Descriptive Statistics
Inferential Statistics and Statistical Significance
Parametric and Nonparametric Statistics
Effect Size
Decision Error and Statistical Power
Hypothesis Testing
Difference-Based and Relationship-Based Analysis
Disaggregating Data
Qualitative Analysis
Sources of Qualitative Data
Coding and Codebooks
Sample Qualitative Analysis Models
Summary
Supplemental Resources
Chapter 14 Reporting, Disseminating, and Utilizing Evaluation Results
Introduction
Reporting Results
The Full Evaluation Report
Other Reporting Mechanisms
Stand-Alone Summaries
A One-Page Bullet Point Summary
A Summary of Conclusions and Recommendations
Feedback Reports
Oral Presentations
Sharing of Raw Data
Developing High-Quality, Accessible Reports and Presentations
Readability
Words Matter
Images Matter
Visually Representing Data
Tables
Figures
Dashboards
Dissemination
Why Disseminate
Dissemination Plans
Using Websites and Social Media
Using Mainstream Media
Keep It Simple
Keep It Interesting
Creative Dissemination Modes
Working With Others
Using Evaluation Results
Summary
Supplemental Resources
Chapter 15 Evaluation as a Business
Introduction
Perspectives on Doing Evaluation as a Business
Ethics
Conflicts of Interest
Protection of Human Participants
Nondiscrimination
Cultural Respect
Business Knowledge and Skills
Marketing
Preparing a Proposal
Making a Budget
Contracts
Contract Components
Intellectual Property
Making a Business Financially Viable
Selecting a Business Entity
Employee
Sole Proprietor
Partner in a Partnership
Limited Liability Company
Corporation/S Corporation
Nonprofit Organization
Bookkeeping and Record Keeping
Developing a Business Plan
Summary
Supplemental Resources
Chapter 16 Interconnections and Practical Implications
Introduction
Objectivity and Bias
Impacts of Bias on Evaluation
Acknowledging Subjectivity and Reducing Bias
Building Cultural Competence
Personalizing a Social Justice Perspective
Reflective Practice and Evaluative Thinking
Applying to Practice
Making Biases Explicit
Infusing Cultural Responsiveness in the Involvement/Engagement of Stakeholders and the Development of Evaluation Questions
Infusing Cultural Responsiveness in Decisions About Evaluation Designs
Infusing Cultural Responsiveness in Decisions About Data Collection and Analysis
Data Collection
Data Analysis
Infusing Cultural Responsiveness in Decisions About Reports and Presentations
Social Justice Evaluation
Politics and Evaluation
Voices From the Field: Advice for New Evaluators
A Final Thought
Supplemental Resources
Appendix A
Appendix B
Glossary
References
Index