Modern Data Architecture on Azure: Design Data-centric Solutions on Microsoft Azure

✍ Scribed by Sagar Lad

• Publisher: Apress
• Year: 2023
• Tongue: English
• Leaves: 216
• Category: Library

✦ Synopsis

This book is an exhaustive guide to designing and implementing data solutions on Azure. It covers the process of managing data from end to end, starting from data collection all the way through transformation, distribution, and consumption.

Modern Data Architecture on Azure begins with an introduction to the fundaments of data management, followed by a demonstration of how to build relational and non-relational data solutions on Azure. Here, you will learn data processing for complex analysis and how to work with CSV and JSON files. Moving forward, you will learn the foundational concepts of big data architecture, along with data management patterns and technology options offered by Azure. From there, you’ll be walked through the data architecture process, including data consortium on Azure, enterprise data governance, and much more. The book culminates with a deep dive into data architecture frameworks with data modeling.

After reading this book, you will have a thorough understanding of data design and analytics using Azure, allowing you to collect and analyze massive amounts of data to optimize business performance, forecast future results, and more.

What Will You Learn

• Understand the fundamentals of data architecture including data management, data handling ethics, data governance, and metadata management
• Analyze and understand business needs to choose the right Azure services and make informed business decisions
• Understand Azure Cloud Data design patterns for relational and non-relational data, batch real-time processing, and ETL/ELT pipelines
• Modernize data architecture using Azure to leverage data and AI to enable digital transformation by securing and optimizing overall data lifecycle management

Who Is This Book For:

Data solution architects, data engineers, and IT consultants who want to gain a better understanding of modern data architecture design and implementation on Azure.

✦ Table of Contents

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Introduction: Fundamentals of Data Management
Introduction to DAMA and DMBOK
Essential Data Concepts
Types of Data
Qualitative Data
Nominal Data
Ordinal Data
Quantitative Data
Discrete Data
Continuous Data
Data Management Principles
The Data Lifecycle
Consistency Models
Data Ingestion Patterns
Data Platform Paradigm
Data Management Principles and Challenges
Preparing a Data Strategy
Defining Roles and Responsibilities
Data Lifecycle Management
Data Quality Measurements
Metadata
Maximizing Data Value for Data-Driven Decisions
Dealing with Substantial Volumes of Data
Siloed and Varied Data Sources
Maintaining the Quality of the Data
Data Integration
Data Governance and Security
Data Automation
Data Management Frameworks
The Strategic Alignment Model
The Amsterdam Information Model
The DAMA DMBOK Framework
The DAMA Wheel
Data Governance
Data Architecture
Data Modeling and Design
Data Storage and Operations
Data Security
Data Integration and Interoperability
Document and Content Management
Reference and Master Data
Data Warehousing and Business Intelligence
Metadata
Data Quality
Understanding the Environmental Factors Hexagon
Understanding the Knowledge Area Context Diagram
Conclusion
Chapter 2: Build Relational and Non-Relational Data Solutions on Azure
Data Integration Using ETL
Data Extraction
Data Transformation
Data Loading
Designing ELT Pipelines Using the Azure Synapse Server
Online Analytical Processing for Complex Analyses
Semantic Data Modeling
Challenges of Using OLAP Solutions
Managing Transaction Data Using OLTP
Managing Non-Relational Data
Key-Value Pair Databases
Column Family Databases
Document Databases
Graph Databases
Handling Time-Series and Free-Form Search Data
Working with CSV and JSON Files for Data Solutions
Conclusion
Chapter 3: Building a Big Data Architecture
Core Components of a Big Data Architecture
Data Ingestion and Processing
Data Analysis
Data Visualization
Data Governance
Using Batch Processing
Azure Synapse Analytics
Azure Data Lake Analytics
Azure Databricks
Azure Data Explorer
Real-Time Processing
Real-Time Data Ingestion
The Lambda Architecture
The Kappa Architecture
Internet of Things (IoT)
Data Mesh Principles and the Logical Architecture
Conclusion
Chapter 4: Data Management Patterns and Technology Choices with Azure
Data Patterns and Trends in Depth
CQRS Pattern
Event Sourcing
Materialized Views
Index Table Pattern
Analytical Store for Big Data Analytics
Azure Synapse Analytics
Azure Databricks
Data Ingestion Process
Data Storage
Data Transformation and Model Training
Analytics
Azure Data Explorer
Building Enterprise Data Lakes and Data Lakehouses
Enterprise Data Lakes
Enterprise Data Lakehouses
Data Pipeline Orchestration
Real-Time Stream Processing in Azure
Conclusion
Chapter 5: Data Architecture Process
Guide to Data Modeling
Conceptual Data Model
Logical Data Model
Physical Data Model
Focus on Business Objectives and its Requirements
Data Lake for Ad Hoc Queries
Enterprise Data Governance: Data Scrambling, Obfuscation, and DataOps
Data Masking Techniques
Data Scrambling
Data Encryption
Data Ageing
Data Substitution
Data Shuffling
Pseudonymization
Master Data Management and Storage Optimization
Master Data Management
Data Encryption Patterns
Conclusion
Chapter 6: Data Architecture Framework Explained
Fundamentals of Data Modeling
The Network Data Model
The Hierarchical Data Model
The Relational Data Model
The Object-Oriented Data Model
The Dimensional Data Model
The Graph Data Model
The Entity Relationship Data Model
The Open Group Architecture Framework
Preliminary Phase
Defining the Architecture Vision
Business Architecture
Information System Architecture
Technology Architecture
Opportunities and Solutions
Migration Planning
Governance Implementation
Architecture Change Management
DAMA DMBOK
The Zachman Framework
Conclusion
Index