Azure Data Factory by Example: Practical Implementation for Data Engineers

✍ Scribed by Richard Swinbank

• Publisher: Apress
• Year: 2021
• Tongue: English
• Leaves: 345
• Edition: 1
• Category: Library

✦ Synopsis

Intermediate user level

✦ Table of Contents

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Creating an Azure Data Factory Instance
Get Started in Azure
Create a Free Azure Account
Explore the Azure Portal
Create a Resource Group
Create an Azure Data Factory
Explore the Azure Data Factory User Experience
Navigation Header Bar
Navigation Sidebar
Link to a Git Repository
Create a Git Repository in Azure Repos
Link the Data Factory to the Git Repository
The ADF UX as a Web-Based IDE
Chapter Review
Key Concepts
For SSIS Developers
Chapter 2: Your First Pipeline
Work with Azure Storage
Create an Azure Storage Account
Explore Azure Storage
Upload Sample Data
Use the Copy Data Tool
Explore Your Pipeline
Linked Services
Datasets
Pipelines
Activities
Integration Runtimes
Factory Resources in Git
Debug Your Pipeline
Run the Pipeline in Debug Mode
Inspect Execution Results
Chapter Review
Key Concepts
For SSIS Developers
Chapter 3: The Copy Data Activity
Prepare an Azure SQL Database
Create the Database
Create Database Objects
Import Structured Data into Azure SQL DB
Create the Basic Pipeline
Create the Database Linked Service and Dataset
Create a DelimitedText File Dataset
Create and Run the Pipeline
Verify the Results
Process Multiple Files
Truncate Before Load
Map Source and Sink Schemas
Create a New Source Dataset
Create a New Pipeline
Configure Schema Mapping
Import Semi-structured Data into Azure SQL DB
Create a JSON File Dataset
Create the Pipeline
Configure Schema Mapping
Set the Collection Reference
The Effect of Schema Drift
Understanding Type Conversion
Transform JSON Files into Parquet
Create a New JSON Dataset
Create a Parquet Dataset
Create and Run the Transformation Pipeline
Performance Settings
Data Integration Unit
Degree of Copy Parallelism
Chapter Review
Key Concepts
Azure Data Factory User Experience (ADF UX)
For SSIS Developers
Chapter 4: Expressions
Explore the Expression Builder
Use System Variables
Enable Storage of Audit Information
Create a New Pipeline
Add New Source Columns
Run the Pipeline
Access Activity Run Properties
Create Database Objects
Add Stored Procedure Activity
Run the Pipeline
Use the Lookup Activity
Create Database Objects
Configure the Lookup Activity
Use Breakpoints
Use the Lookup Value
Update the Stored Procedure Activity
Run the Pipeline
User Variables
Create a Variable
Set a Variable
Use the Variable
Array Variables
Concatenate Strings
Infix Operators
String Interpolation
Escaping @
Chapter Review
Key Concepts
For SSIS Developers
Chapter 5: Parameters
Set Up an Azure Key Vault
Create a Key Vault
Create a Key Vault Secret
Grant Access to the Key Vault
Create a Key Vault ADF Linked Service
Create a New Storage Account Linked Service
Use Dataset Parameters
Create a Parameterized Dataset
Use the Parameterized Dataset
Reuse the Parameterized Dataset
Use Linked Service Parameters
Create a Parameterized Linked Service
Increase Dataset Reusability
Use the New Dataset
Why Parameterize Linked Services?
Use Pipeline Parameters
Create a Parameterized Pipeline
Run the Parameterized Pipeline
Use the Execute Pipeline Activity
Parallel Execution
Global Parameters
Chapter Review
Key Concepts
For SSIS Developers
Chapter 6: Controlling Flow
Create a Per-File Pipeline
Use Activity Dependency Conditions
Explore Dependency Condition Interactions
Understand the Skipped Condition
Understand the Failed Condition
Combine Conditions
Create Dependencies on Multiple Activities
Understand the Completion Condition
Understand Pipeline Outcome
Raise Errors
Use Conditional Activities
Divert Error Rows
Load Error Rows
Create a New Sink Dataset
Revise the Source Dataset
Use the If Condition Activity
Run the Pipeline
Understand the Switch Activity
Use Iteration Activities
Use the Get Metadata Activity
Use the ForEach Activity
Ensure Parallelizability
Understand the Until Activity
Chapter Review
Key Concepts
For SSIS Developers
Chapter 7: Data Flows
Build a Data Flow
Enable Data Flow Debugging
Add a Data Flow Transformation
Use the Filter Transformation
Use the Lookup Transformation
Add a Lookup Data Stream
Add the Lookup Transformation
Use the Derived Column Transformation
Use the Select Transformation
Use the Sink Transformation
Execute the Data Flow
Create a Pipeline to Execute the Data Flow
Inspect Execution Output
Persist Loaded Data and Log Completion
Maintain a Product Dimension
Create a Dimension Table
Create Supporting Datasets
Build the Product Maintenance Data Flow
Use Locals
Use the Aggregate Transformation
Use the Exists Transformation
Execute the Dimension Data Flow
Chapter Review
Key Concepts
For SSIS Developers
Chapter 8: Integration Runtimes
Azure Integration Runtime
Inspect the AutoResolveIntegrationRuntime
Create a New Azure Integration Runtime
Databricks Cluster TTL
Controlling the Geography of Data Movement
Use the New Azure Integration Runtime
Identify the Copy Data Effective Integration Runtime
Revise the Sink Integration Runtime
Self-Hosted Integration Runtime
Create a Shared Data Factory
Create a Self-Hosted Integration Runtime
Link to a Self-Hosted Integration Runtime
Use the Self-Hosted Integration Runtime
Create a Linked Service Using the Shared Self-Hosted IR
Create a File System Dataset
Copy Data Using the File System Dataset
Azure-SSIS Integration Runtime
Create an Azure-SSIS Integration Runtime
Deploy SSIS Packages to the Azure-SSIS IR
Run an SSIS Package in ADF
Stop the Azure-SSIS IR
Chapter Review
Key Concepts
For SSIS Developers
Chapter 9: Power Query in ADF
Create a Power Query Mashup
Explore the Power Query Editor
Wrangle Data
Run the Power Query Activity
Chapter Review
Chapter 10: Publishing to ADF
Publish to Your Factory Instance
Trigger a Pipeline from the ADF UX
Publish Factory Resources
Inspect Published Pipeline Run Outcome
Publish to Another Data Factory
Prepare a Production Environment
Create the Production Factory
Grant Access to the Self-Hosted Integration Runtime
Export ARM Template from Your Development Factory
Import ARM Template into Your Production Factory
Understand Deployment Parameters
Automate Publishing to Another Factory
Create a DevOps Service Connection
Create an Azure DevOps Pipeline
Create a YAML Pipeline File
Create an Azure DevOps Pipeline Using the YAML File
Add the Factory Deployment Task
Trigger an Automatic Deployment
Feature Branch Workflow
Azure Data Factory Utilities
Publish Resources as JSON
Deploy ADF Pipelines Using PowerShell
Resource Dependencies
Chapter Review
Chapter 11: Triggers
Use a Schedule Trigger
Create a Schedule Trigger
Reuse a Trigger
Inspect Trigger Definitions
Publish the Trigger
Monitor Trigger Runs
Deactivate the Trigger
Advanced Recurrence Options
Use an Event-Based Trigger
Register the Event Grid Resource Provider
Create an Event-Based Trigger
Cause the Trigger to Run
Trigger-Scoped System Variables
Use a Tumbling Window Trigger
Prepare Data
Create a Windowed Copy Pipeline
Create a Tumbling Window Trigger
Monitor Trigger Runs
Advanced Features
Publishing Triggers Automatically
Triggering Pipelines Programmatically
Chapter Review
Key Concepts
For SSIS Developers
Chapter 12: Monitoring
Generate Factory Activity
Inspect Factory Logs
Inspect Trigger Runs
Inspect Pipeline Runs
Add Metadata to the Log
Add a Pipeline Annotation
Add an Activity User Property
Inspect Pipeline Annotations in the Log
Inspect User Properties in the Log
Inspect Factory Metrics
Export Logs and Metrics
Create a Log Analytics Workspace
Configure Diagnostic Settings
Inspect Logs in Blob Storage
Use the Log Analytics Workspace
Query Logs
Use a Log Analytics Workbook
Receive Alerts
Configure Metric-Based Alerts
Configure Log-Based Alerts
Deactivate ADF Triggers
Chapter Review
Key Concepts
For SSIS Developers
Index