Front Cover
How do you know if you are Making a Difference?: A Practical Handbook for Public Service Organisations
Copyright information
Dedication
Table of contents
List of figures and tables
Acknowledgments
Preface
1 Why is it so hard to know if you are making a difference?
Why it is so hard to know the difference you are making
The benefits of investing in knowing the difference you are making
What you need to go on this journey
A strategic approach
Embedding learning and evaluation into organisational practice
Public service improvement as practice
Meanings: ideas and aspirations
Competencies: skills, knowledge, technique
Materials: objects, technologies
Our approach to understanding if you are making a difference
Anchored in practice and the practitioners of public services
Overview of this book
Try it yourself
2 Why complexity thinking can help you understand public services
A shift in thinking for complexity-informed approaches
Working with complex systems
Doing the right thing, not the same thing every time
Ways of managing in complexity
Navigating through complexity
People at the heart of complex change
People experience the system in different ways
Co-production and participatory approaches
The โwhoโ in data and feedback
Leadership for complex work
Evaluating in complexity
The central concept of โcontributionโ
Contribution analysis
Appropriately different evaluation every time
How our approach helps to work with complexity
Building blocks of our complexity-informed approach
Try it yourself
Questions for exploring the complexity of your work
3 What data and evidence do you need to see what difference you are making?
What is included when we talk about data, evidence and feedback?
Good data and evidence for complex change
Letโs measure what matters
Data cultures and how they affect organisations and initiatives
Making the case for qualitative data
Capturing evidence from practice โ the power of reflection
What is happening when people โuseโ data?
Getting started on a data improvement journey
Try it yourself
4 Owning your initiativeโs outcomes and impacts
Navigating the landscape of outcomes and impacts
Drivers of outcome- or impact-based approaches
The pressures of working to multiple outcome frameworks
Escaping multiple outcome demands by owning your outcomes
An introduction to outcome thinking
What we mean by an outcome
Outcomes (or impacts) as a framework for action
Outcomes at different levels
Some common myths about outcomes
A new way of working with outcomes
Owning your outcomes: what this means in practice
Try it yourself
5 Embrace the complex context
Why context matters
The benefits of taking time to work with context
A practical framework to help understand context
ISM in action
Justifying resources in ethnic minority health
Setting realistic expectations for changing residential care
Doing the right thing โ better dementia training for carers
Linking context and outcomes with risks and assumptions
What are risks and assumptions?
Bringing context and risk together for action
How to test assumptions and monitor risks
Conclusions
Try it yourself
6 Clarify the change you want to see
Thinking about whether and how something makes a difference: an introduction to theories of change and outcome mapping
You know where you are with an outcome map
Our outcome mapping approach
The key components of outcome or impact maps
What we do
Who with
How they feel
What they learn and gain
What they do differently
What difference does this make?
Itโs not too late!
What is different about this approach
Process focus
How is time represented?
A map of the complex world
A flexible framework
The role of pathways in understanding change
Why are pathways helpful?
Tips for making the best of pathways
Three ways to organise pathways
Including emotions
Outcome mapping at different levels: project, programme, organisation or partnership
Outcome mapping for programmes
Outcome mapping for organisations
Outcome mapping for partnerships
Bringing an outcome map to life with risks and assumptions
Considering risks and assumptions for an outcome map or pathway
The art of outcome mapping
Conclusions
Try it yourself
7 Get going on your data, evidence and feedback improvement journey
Improving approaches to data, evidence and feedback
Define the appropriate scale for data collection
Focus on usability
Mixed data is best
How to undertake a data audit
Step 1: What data already exists about the difference this makes?
Step 2: What does or doesnโt your existing data tell you?
Step 3: What gaps in the data are there for each pathway?
How to fill gaps in understanding
Outcome maps and pathways as data collection guides
Practical data collection techniques
Formalise insights โ reflective impact log
Bridging the gap between data about people and organisation-wide impact
Case note sampling
Individual outcome recording
Getting feedback when itโs difficult
Rapid feedback techniques
Quick feedback form
Flip charts
Smiley faces
Images
Informal feedback or โthank bankโ
Do it your way
Feedback in action
Health Technology Wales: web interface feedback loop
Scottish Social Services Council: reflective impact log
East Renfrewshire peer support project: case note audit
About Dementia: fast feedback
Conclusion
Try it yourself
8 Tracking progress towards outcomes and impacts
Getting started with data analysis
What is data analysis?
The process of qualitative analysis
Questions
Preliminary analysis
Developing analysis
Maximising rigour
A practical framework for assessing and tracking the difference you make
A process for analysing data, evidence and feedback
A holistic view of progress
Visualising progress and confidence in evidence
Three levels to help shape analysis
Level 1: Can this initiative make the difference it hopes to?
Level 2: How does this initiative work: what helps and what gets in the way?
Level 3: To what extent does this initiative make a difference?
Conclusion
Try it yourself
9 Telling the story of the difference your work makes
Reporting in an outcome- and impact-focused way
Reporting for different audiences and purposes
Reporting for internal learning
Reporting for improvement
Reporting for accountability
Reporting for raising awareness and communication
Reporting to funders
Using an outcome map as a framework for reporting
Outcome maps and pathways link activities and outcomes
A strong foundation of evidence
A compelling narrative
Focus on the most important contribution made
Practical tips for great reporting
Components of an outcome- or impact-focused report
Tips for robust reporting
Showing โenoughโ evidence for the work
Reporting well on different parts of a contribution story
Reporting on activities: โwhat we doโ
Reporting on engagement: โwho withโ
Reporting on reactions: โhow they feelโ
Reporting on โwhat they learn and gainโ
Reporting on โwhat they do differentlyโ
Reporting on โwhat difference does it make?โ
Being honest about challenges
Bring it to life
Taking reporting forward
Try it yourself
10 Using this approach in different contexts and sectors
Organisations and settings using this approach
Outcome and impact mapping and tracking
Embedding outcome and impact tracking as a change process
Facing common issues
Are these the right outcomes?
Are activities aligned to outcomes?
Where is the data?
Getting the skills and capacity
Drivers for embedding outcome and impact evaluation
Practical tips for your outcome- and impact-evaluation journey
Case studies: embedding outcome and impact tracking in organisations
Outcome tracking for new and innovative ways of working
Outcome tracking across a small organisation
Outcome and impact tracking for a research organisation
Conclusion
11 Becoming an outcome- and impact-focused organisation
Advice from other organisations on the journey
References
Index
Back Cover