Dask: The Definitive Guide - Scalable Python Data Science with Dask (Early Release 1)

✍ Scribed by Matthew Rocklin, Matthew Powers, Richard Pelgrim

• Publisher: O’Reilly Media, Inc.
• Year: 2022
• Tongue: English
• Leaves: 105
• Edition: 1
• Category: Library

✦ Synopsis

The burgeoning volume and complexity of data make scalability and reliability increasingly challenging issues. But while modern systems contain multicore CPUs and GPUs that have the potential for parallel computing, many Python tools weren't designed to leverage this parallelism. Using Dask to parallelize Python workflows delivers a competitive advantage by reducing turnaround time, freeing you to work on more interesting or complex data problems.

With this essential guide at your side, you'll be able to:

Deploy Dask on the cloud or on-prem
Scale your Python code to bigger datasets and CPU-intensive workflows
Speed up data pipelines that often take weeks or months to execute
Overcome the limits of serial computing on your local machine (or system of machines)
Use the examples provided to scale your workflows, whether you're working with NumPy, pandas, scikit-learn, PyTorch, XGBoost, or other tools
Develop a specialized data science library that leverages parallel and distributed computing
Scale computations to a cluster of machines and to the cloud securely and efficiently

✦ Table of Contents

1. Understanding the Architecture of Dask DataFrames
   pandas Architecture
   pandas Limitations
   How Dask DataFrames Differ from pandas
   Example illustrating Dask DataFrame Architectural Components
   Lazy Execution
   Dask DataFrame Divisions
   pandas vs. Dask DataFrame on Larger than RAM Datasets
   Scaling Up vs Scaling Out
   Summary
2. How to Work with Dask DataFrames
   Reading Data into a Dask DataFrame
   Read a single file into a Dask DataFrame
   Read multiple files into a Dask DataFrame
   Working with partitioned data
   Setting Partition Size
   Inspecting Data Types
   Reading Remote Data from the Cloud
   Processing Data with Dask DataFrames
   Converting to Parquet files
   Materializing results in memory with compute
   Materializing results in memory with persist()
   Repartitioning Dask DataFrames
   Filtering Dask DataFrames
   Setting the Index
   Joining Dask DataFrames
   Mapping Custom Functions
   groupby aggregations
   Memory usage
   Tips on managing memory
   Converting to number columns with to_numeric
   Vertically union Dask DataFrames
   Writing Data with Dask DataFrames
   File Compression
   to_csv: single_file
   to_parquet: engine
   to_parquet: partition_on
   Other Keywords
   Summary