Driving Data Quality with Data Contracts: A comprehensive guide to building reliable, trusted, and effective data platforms

✍ Scribed by Andrew Jones

• Publisher: Packt Publishing
• Tongue: English
• Leaves: 207
• Category: Library

✦ Synopsis

Everything you need to know to apply data contracts and build a truly data-driven organization that harnesses quality data to deliver tangible business value

Purchase of the print or Kindle book includes a free PDF eBook.

Key Features

• Understand data contracts and their power to resolving the problems in contemporary data platforms
• Learn how to design and implement a cutting-edge data platform powered by data contracts
• Access practical guidance from the pioneer of data contracts to get expert insights on effective utilization

Book Description

Despite the passage of time and the evolution of technology and architecture, the challenges we face in building data platforms persist. Our data often remains unreliable, lacks trust, and fails to deliver the promised value.

With Driving Data Quality with Data Contracts, you'll discover the potential of data contracts to transform how you build your data platforms, finally overcoming these enduring problems. You'll learn how establishing contracts as the interface allows you to explicitly assign responsibility and accountability of the data to those who know it best—the data generators—and give them the autonomy to generate and manage data as required. The book will show you how data contracts ensure that consumers get quality data with clearly defined expectations, enabling them to build on that data with confidence to deliver valuable analytics, performant ML models, and trusted data-driven products.

By the end of this book, you'll have gained a comprehensive understanding of how data contracts can revolutionize your organization's data culture and provide a competitive advantage by unlocking the real value within your data.

What you will learn

• Gain insights into the intricacies and shortcomings of today's data architectures
• Understand exactly how data contracts can solve prevalent data challenges
• Drive a fundamental transformation of your data culture by implementing data contracts
• Discover what goes into a data contract and why it's important
• Design a modern data architecture that leverages the power of data contracts
• Explore sample implementations to get practical knowledge of using data contracts
• Embrace best practices for the successful deployment of data contracts

Who this book is for

If you're a data engineer, data leader, architect, or practitioner thinking about your data architecture and looking to design one that enables your organization to get the most value from your data, this book is for you. Additionally, staff engineers, product managers, and software engineering leaders and executives will also find valuable insights.

Table of Contents

1. A Brief History of Data Platforms
2. Introducing Data Contracts
3. How to Get Adoption in Your Organization
4. Bringing Data Consumers and Generators Closer Together
5. Embedding Data Governance
6. What Makes Up a Data Contract
7. A Contract-Driven Data Architecture
8. A Sample Implementation
9. Implementing Data Contracts In Your Organization
10. Data Contracts in Practice

✦ Table of Contents

Cover
Title Page
Copyright and Credits
Dedication
Foreword
Contributors
Table of Contents
Preface
Part 1: Why Data Contracts?
Chapter 1: A Brief History of Data Platforms
The enterprise data warehouse
The big data platform
The modern data stack
The state of today’s data platforms
The lack of expectations
The lack of reliability
The lack of autonomy
The ever-increasing use of data in business-critical applications
Summary
Further reading
Chapter 2: Introducing Data Contracts
What is a data contract?
An agreed interface between the generators of data, and its consumers
Setting expectations around that data
Defining how the data should be governed
Facilitating the explicit generation of quality data
The four principles of data contracts
When to use data contracts
Data contracts and the data mesh
Domain ownership
Data as a product
Self-serve data platform
Federated computational governance
Data contracts enable a data mesh
Summary
Further reading
Part 2: Driving Data Culture Change with Data Contracts
Chapter 3: How to Get Adoption in Your Organization
Using data contracts to change an organization
Articulating the value of your data
Building data products
What is a data product?
Adopting a data product mindset
Designing a data product
Walking through an example of a data product
Summary
Further reading
Chapter 4: Bringing Data Consumers and Generators Closer Together
Who is a consumer, and who is a generator?
Data consumers
Data generators
Assigning responsibility and accountability
Feeding data back to the product teams
Managing the evolution of data
Summary
Further reading
Chapter 5: Embedding Data Governance
Why we need data governance
The requirements of data governance
How data governance programs are typically applied
Promoting data governance through data contracts
Assigning responsibility for data governance
Responsibilities of the data generators
Introducing the data architecture council
Working together to implement federated data governance
Summary
Further reading
Part 3: Designing and Implementing a Data Architecture Based on Data Contracts
Chapter 6: What Makes Up a Data Contract
The schema of a data contract
Defining a schema
Using a schema registry as the source of truth
Evolving your data over time
Evolving your schemas
Migrating your consumers
Defining the governance and controls
Summary
Further reading
Chapter 7: A Contract-Driven Data Architecture
A step-change in building data platforms
Building generic data tooling
Introducing a data infrastructure team
A case study from GoCardless in promoting autonomy
Promoting autonomy through decentralization
Introducing the principles of a contract-driven data architecture
Automation
Guidelines and guardrails
Consistency
Providing self-served data infrastructure
Summary
Further reading
Chapter 8: A Sample Implementation
Technical requirements
Creating a data contract
Providing the interfaces to the data
Introducing IaC
Creating the interfaces from the data contract
Creating libraries for data generators
Populating a central schema registry
Registering a schema with the Confluent schema registry
Managing schema evolution
Implementing contract-driven tooling
Summary
Further reading
Chapter 9: Implementing Data Contracts in Your Organization
Getting started with data contracts
The ability to define a data contract
The ability to provision an interface for the data for consumers to query
The ability of generators to write data to the interface
Migrating to data contracts
Discovering data contracts
What is a data catalog?
Why are data catalogs important for discovering data contracts?
What is data lineage?
Why is data lineage important for data contracts?
Building a mature data contracts-backed data culture
Summary
Further reading
Chapter 10: Data Contracts in Practice
Designing a data contract
Identifying the purpose
Considering the trade-offs
Defining the data contract
Deploying the data contract
Monitoring and enforcing data contracts
The data contract’s definition
The quality of the data
The performance and dependability of the data
Data contract publishing patterns
Writing directly to the interface
Materialized views on CDC
The transactional outbox pattern
The listen-to-yourself pattern
Summary
Further reading
Index
About Packt
Other Books You May Enjoy