Distributed Computing with Python

✍ Scribed by Pierfederici, Francesco

• Publisher: Packt Publishing
• Year: 2016
• Tongue: English
• Leaves: 171
• Category: Library

✦ Synopsis

Key Features You'll learn to write data processing programs in Python that are highly available, reliable, and fault tolerant Make use of Amazon Web Services along with Python to establish a powerful remote computation system Train Python to handle data-intensive and resource hungry applications Book DescriptionCPU-intensive data processing tasks have become crucial considering the complexity of the various big data applications that are used today. Reducing the CPU utilization per process is very important to improve the overall speed of applications.This book will teach you how to perform parallel execution of computations by distributing them across multiple processors in a single machine, thus improving the overall performance of a big data processing task. We will cover synchronous and asynchronous models, shared memory and file systems, communication between various processes, synchronization, and more.What You Will Learn Get an introduction to parallel and distributed computing See synchronous and asynchronous programming Explore parallelism in Python Distributed application with Celery Python in the Cloud Python on an HPC cluster Test and debug distributed applications About the AuthorFrancesco Pierfedericiis a software engineer who loves Python. He has been working in the fields of astronomy, biology, and numerical weather forecasting for the last 20 years.He has built large distributed systems that make use of tens of thousands of cores at a time and run on some of the fastest supercomputers in the world. He has also written a lot of applications of dubious usefulness but that are great fun. Mostly, he just likes to build things.Table of Contents An Introduction to Parallel and Distributed Computing Asynchronous Programming Parallelism in Python Distributed Applications – with Celery Python in the Cloud Python on an HPC Cluster Testing and Debugging Distributed Applications The Road Ahead

✦ Table of Contents

Cover......Page 1
Copyright......Page 3
Credits......Page 4
About the Author......Page 5
About the Reviewer......Page 6
www.PacktPub.com......Page 7
Table of Contents......Page 8
Preface......Page 10
Chapter 1: An Introduction to Parallel and Distributed Computing......Page 16
Parallel computing......Page 17
Distributed computing......Page 19
Shared memory versus distributed memory......Page 21
Amdahl's law......Page 24
Summary......Page 27
Chapter 2: Asynchronous Programming......Page 28
Coroutines......Page 31
An asynchronous example......Page 37
Summary......Page 43
Chapter 3: Parallelism in Python......Page 44
Multiple threads......Page 45
Multiple processes......Page 52
Multiprocess queues......Page 57
Closing thoughts......Page 59
Summary......Page 60
Establishing a multimachine environment......Page 62
Installing Celery......Page 64
Testing the installation......Page 67
A tour of Celery......Page 70
More complex Celery applications......Page 72
Celery in production......Page 80
Celery alternatives – Python-RQ......Page 82
Celery alternatives – Pyro......Page 85
Summary......Page 92
Cloud computing and AWS......Page 94
Creating an AWS account......Page 95
Creating an EC2 instance......Page 105
Storing data in Amazon S3......Page 114
Amazon elastic beanstalk......Page 118
Creating a private cloud......Page 119
Summary......Page 120
Your typical HPC cluster......Page 122
Job schedulers......Page 124
Running a Python job using HTCondor......Page 126
Running a Python job using PBS......Page 138
Debugging......Page 143
Summary......Page 144
Chapter 7: Testing and Debugging Distributed Applications......Page 146
Common problems – clocks and time......Page 147
Common problems – software environments......Page 149
Common problems – permissions and environments......Page 150
Common problems – the availability of hardware resources......Page 151
Challenges – the development environment......Page 155
A useful strategy – logging everything......Page 156
A useful strategy – simulating components......Page 158
Summary......Page 159
Chapter 8: The Road Ahead......Page 160
The first two chapters......Page 161
The tools......Page 162
The cloud and the HPC world......Page 163
Debugging and monitoring......Page 165
Where to go next......Page 166
Index......Page 168