Measuring Data Quality for Ongoing Improvement: A Data Quality Assessment Framework

✍ Scribed by Laura Sebastian-Coleman

• Publisher: Morgan Kaufmann
• Year: 2013
• Tongue: English
• Leaves: 404
• Edition: 1
• Category: Library

✦ Synopsis

The Data Quality Assessment Framework shows you how to measure and monitor data quality, ensuring quality over time. You'll start with general concepts of measurement and work your way through a detailed framework of more than three dozen measurement types related to five objective dimensions of quality: completeness, timeliness, consistency, validity, and integrity. Ongoing measurement, rather than one time activities will help your organization reach a new level of data quality. This plain-language approach to measuring data can be understood by both business and IT and provides practical guidance on how to apply the DQAF within any organization enabling you to prioritize measurements and effectively report on results. Strategies for using data measurement to govern and improve the quality of data and guidelines for applying the framework within a data asset are included. You'll come away able to prioritize which measurement types to implement, knowing where to place them in a data flow and how frequently to measure. Common conceptual models for defining and storing of data quality results for purposes of trend analysis are also included as well as generic business requirements for ongoing measuring and monitoring including calculations and comparisons that make the measurements meaningful and help understand trends and detect anomalies. Demonstrates how to leverage a technology independent data quality measurement framework for your specific business priorities and data quality challenges Enables discussions between business and IT with a non-technical vocabulary for data quality measurement Describes how to measure data quality on an ongoing basis with generic measurement types that can be applied to any situation

✦ Table of Contents

Front Cover......Page 1
Measuring Data Quality for Ongoing Improvement......Page 4
Copyright Page......Page 5
Contents......Page 8
Acknowledgments......Page 24
Foreword......Page 26
Author Biography......Page 28
Data Quality Measurement: the Problem we are Trying to Solve......Page 30
Definitions of Data Quality......Page 31
The Criticality of Metadata and Explicit Knowledge......Page 32
DQAF: the Data Quality Assessment Framework......Page 33
Section Two: DQAF Overview......Page 34
Section Five: Data Quality Strategy......Page 35
Intended Audience......Page 36
What Measuring Data Quality for Ongoing Improvement Does Not Do......Page 37
Why I Wrote Measuring Data Quality for Ongoing Improvement......Page 38
1: Concepts and Definitions......Page 42
Data......Page 44
Data as Representation......Page 45
The Implications of Data’s Semiotic Function......Page 47
Semiotics and Data Quality......Page 49
Data as a Product......Page 52
Data as Input to Analyses......Page 53
Data and Expectations......Page 54
Information......Page 55
Concluding Thoughts......Page 56
Enterprise or Organization......Page 58
IT and the Business......Page 59
Data Consumers......Page 60
Data Stewards and Data Stewardship......Page 61
Data Ownership and Data Governance......Page 62
IT, the Business, and Data Owners, Redux......Page 63
Data Quality Program Team......Page 64
Systems and System Design......Page 65
Concluding Thoughts......Page 66
Data Management......Page 68
Database, Data Warehouse, Data Asset, Dataset......Page 69
Source System, Target System, System of Record......Page 70
Data Models......Page 71
Types of Data Models......Page 72
Metadata......Page 73
Metadata as Explicit Knowledge......Page 76
Data Chain and Information Life Cycle......Page 77
Concluding Thoughts......Page 78
Data Quality......Page 80
Data Quality Dimensions......Page 81
Measurement......Page 82
Measurement as Data......Page 83
Data Quality Measurement and the Business/IT Divide......Page 84
Measurements must be Comprehensible and Interpretable......Page 85
Measurements must be Reproducible......Page 86
Data Quality Assessment......Page 87
Data Quality Dimensions, DQAF Measurement Types, Specific Data Quality Metrics......Page 88
Reasonability Checks......Page 90
Data Quality Thresholds......Page 91
Process Controls......Page 93
Concluding Thoughts......Page 94
2: DQAF Concepts and Measurement Types......Page 96
The Problem the DQAF Addresses......Page 98
Data Quality Expectations and Data Management......Page 99
The Scope of the DQAF......Page 101
DQAF Quality Dimensions......Page 102
Validity......Page 103
The Question of Accuracy......Page 104
Metadata Requirements......Page 105
Objects of Measurement and Assessment Categories......Page 106
Functions in Measurement: Collect, Calculate, Compare......Page 107
Concluding Thoughts......Page 110
Consistency of the Data Model......Page 112
Inspecting the Condition of Data upon Receipt......Page 113
Assessing the Validity of Data Content......Page 115
Assessing the Consistency of Data Content......Page 117
Comments on the Placement of In-line Measurements......Page 119
Periodic Measurement of Cross-table Content Integrity......Page 122
Assessing Overall Database Content......Page 123
The Measurement Types: Consolidated Listing......Page 124
Concluding Thoughts......Page 132
Purpose......Page 134
Metadata: Knowledge before Assessment......Page 136
Input to Initial Assessments......Page 138
Data Expectations......Page 139
Column Property Profiling......Page 141
Date Data......Page 142
The Presence of Invalid Values......Page 143
Columns with High Cardinality Where Low Cardinality is Expected......Page 144
Multiple Data Format Patterns in One Column......Page 145
Incomplete Referential Relationships......Page 146
Missing or Unexpected Dependency Relationships and Rules......Page 148
Differences in Granularity and Precision......Page 149
Profiling an Existing Data Asset......Page 150
Deliverables from Initial Assessment......Page 151
Concluding Thoughts......Page 152
Data Quality Improvement Efforts......Page 154
Measurement in Improvement Projects......Page 155
The Case for Ongoing Measurement......Page 158
Example: Health Care Data......Page 160
Inputs for Ongoing Measurement......Page 162
Automation......Page 164
Controls......Page 165
Periodic Measurement......Page 166
In-Line versus Periodic Measurement......Page 167
Concluding Thoughts......Page 168
4: Applying the DQAF to Data Requirements......Page 170
Business Requirements......Page 174
Data Quality Requirements and Expected Data Characteristics......Page 177
Data Quality Requirements and Risks to Data......Page 180
Factors Influencing Data Criticality......Page 181
Specifying Data Quality Metrics......Page 182
Subscriber Birth Date Example......Page 183
Additional Examples......Page 187
Concluding Thoughts......Page 190
Asking Questions......Page 192
Understanding the Project......Page 193
Learning about Source Systems......Page 194
Source Goals and Source Data Consumers......Page 195
Individual Data Attributes and Rules......Page 196
Your Data Consumers’ Requirements......Page 197
The Condition of the Data......Page 198
Measurement Specification Process......Page 199
Concluding Thoughts......Page 203
5: A Strategic Approach to Data Quality......Page 204
The Concept of Strategy......Page 206
Systems Strategy, Data Strategy, and Data Quality Strategy......Page 207
Data Quality Strategy and Data Governance......Page 209
Decision Points in the Information Life Cycle......Page 210
General Considerations for Data Quality Strategy......Page 211
Concluding Thoughts......Page 212
Purpose......Page 214
Assessing Management Commitment......Page 217
Directive 2: Treat Data as an Asset......Page 218
Directive 3: Apply Resources to Focus on Quality......Page 219
Assessing Readiness to Commission a Data Quality Team......Page 220
Assessing the Condition of Explicit Knowledge and Knowledge Sharing......Page 221
Directive 5: Treat Data as a Product of Processes that can be Measured and Improved......Page 222
Directive 6: Recognize Quality is Defined by Data Consumers......Page 223
Assessing How Data Consumers Define Data Quality......Page 224
Directive 7: Address the Root Causes of Data Problems......Page 225
Directive 8: Measure Data Quality, Monitor Critical Data......Page 227
Assessing Organizational Readiness for Ongoing Measurement and Monitoring......Page 228
Assessing Options for Accountability......Page 229
Directive 11: Data Needs and Uses will Evolve—Plan for Evolution......Page 230
Developing a Plan for Evolution......Page 231
Building a Culture Focused on Data Quality......Page 232
Concluding Thoughts: Using the Current State Assessment......Page 233
6: The DQAF in Depth......Page 234
Facets of the DQAF Measurement Types......Page 235
Functions in Measurement: Collect, Calculate, Compare......Page 238
Calculating Measurement Data......Page 240
Statistics......Page 242
Measures of Variability......Page 243
The Control Chart: A Primary Tool for Statistical Process Control......Page 246
The DQAF and Statistical Process Control......Page 247
Concluding Thoughts......Page 248
Metric Definition and Measurement Result Tables......Page 250
Common Key Fields......Page 252
Optional Fields......Page 253
Denominator Fields......Page 254
Automated Thresholds......Page 256
Emergency Thresholds......Page 257
Additional System Requirements......Page 258
Concluding Thoughts......Page 259
Facets of the DQAF......Page 260
Organization of the Chapter......Page 262
Measurement Methodology......Page 265
Definition......Page 266
Support Processes and Skills......Page 267
Definition......Page 268
Definition......Page 269
Business Concerns......Page 270
Support Processes and Skills......Page 271
Measurement Logical Data Model......Page 272
Business Concerns......Page 273
Definition......Page 274
Business Concerns......Page 275
Business Concerns......Page 276
Measurement Logical Data Model......Page 277
Measurement Methodology......Page 278
Programming......Page 279
Support Processes and Skills......Page 280
Support Processes and Skills......Page 281
Definition......Page 282
Programming......Page 283
Measurement Logical Data Model......Page 284
Support Processes and Skills......Page 285
Measurement Logical Data Model......Page 286
Support Processes and Skills......Page 288
Programming......Page 289
Measurement Logical Data Model......Page 290
Business Concerns......Page 291
Definition......Page 292
Measurement Methodology......Page 293
Measurement Methodology......Page 294
Support Processes and Skills......Page 295
Business Concerns......Page 296
Measurement Logical Data Model......Page 297
Measurement Methodology......Page 298
Measurement Logical Data Model......Page 299
Business Concerns......Page 300
Measurement Logical Data Model......Page 301
Measurement Methodology......Page 302
Programming......Page 303
Measurement Logical Data Model......Page 304
Business Concerns......Page 305
Measurement Logical Data Model......Page 306
Business Concerns......Page 307
Definition......Page 308
Support Processes and Skills......Page 310
Definition......Page 311
Programming......Page 312
Definition......Page 313
Measurement Logical Data Model......Page 314
Business Concerns......Page 315
Support Processes and Skills......Page 316
Measurement Logical Data Model......Page 317
Definition......Page 318
Other Facets......Page 319
Programming......Page 320
Measurement Logical Data Model......Page 321
Business Concerns......Page 322
Support Processes and Skills......Page 323
Measurement Logical Data Model......Page 324
Business Concerns......Page 325
Support Processes and Skills......Page 326
Definition......Page 327
Support Processes and Skills......Page 328
Definition......Page 329
Programming......Page 330
Support Processes and Skills......Page 331
Support Processes and Skills......Page 332
Programming......Page 333
Definition......Page 334
Definition......Page 335
Definition......Page 336
Business Concerns......Page 337
Business Concerns......Page 338
Concluding Thoughts: Know Your Data......Page 339
Glossary......Page 342
Bibliography......Page 354
Index......Page 360
Appendix A: Measuring the Value of Data......Page 366
Richard Wang’s and Diane Strong’s Data Quality Framework, 1996......Page 370
Thomas Redman’s Dimensions of Data Quality, 1996......Page 371
Larry English’s Information Quality Characteristics and Measures, 1999......Page 373
Purpose......Page 376
High-Level Assessment......Page 377
Detailed Assessment......Page 378
Quality of Definitions......Page 382
Summary......Page 384
Purpose......Page 386
Limitations of the Communications Model of Information Quality......Page 387
Error, Prediction, and Scientific Measurement......Page 388
What Do We Learn from Ivanov?......Page 389
Ivanov’s Concept of the System as Model......Page 390
A Brief History of Quality Improvement......Page 392
Walter Shewhart......Page 393
Joseph Juran......Page 394
Philip Crosby......Page 395
Process Flowcharts......Page 396
Plan, Do, Study, Act......Page 397
Define, Measure, Analyze, Improve, Control......Page 399
Implications for Data Quality......Page 400
Limitations of the Data as Product Metaphor......Page 401
Data as a Product Redux......Page 402
Concluding Thoughts: Building Quality in Means Building Knowledge in......Page 403

✦ Subjects

Библиотека;Компьютерная литература;Алгоритмы и структуры данных;