Advanced Python Programming: Build high performance, concurrent, and multi-threaded apps with Python using proven design patterns

✍ Scribed by Lanaro, Dr Gabriele;Nguyen, Quan;Kasampalis, Sakis

• Publisher: Packt Publishing
• Year: 2019
• Tongue: English
• Leaves: 652
• Category: Library

✦ Table of Contents

Cover......Page 1
Title Page......Page 2
Copyright......Page 3
About Packt......Page 4
Contributors......Page 5
Table of Contents......Page 7
Preface......Page 18
Chapter 1: Benchmarking and Profiling......Page 26
Designing your application......Page 27
Writing tests and benchmarks......Page 33
Timing your benchmark......Page 35
Better tests and benchmarks with pytest-benchmark......Page 38
Finding bottlenecks with cProfile......Page 41
Profile line by line with line_profiler......Page 46
Optimizing our code......Page 48
The dis module......Page 50
Profiling memory usage with memory_profiler......Page 51
Summary......Page 54
Chapter 2: Pure Python Optimizations......Page 55
Useful algorithms and data structures......Page 56
Lists and deques......Page 57
Dictionaries......Page 59
Building an in-memory search index using a hash map......Page 61
Sets......Page 63
Heaps......Page 64
Tries......Page 66
Caching and memoization......Page 68
Joblib......Page 71
Comprehensions and generators......Page 72
Summary......Page 74
Chapter 3: Fast Array Operations with NumPy and Pandas......Page 75
Creating arrays......Page 76
Accessing arrays......Page 78
Broadcasting......Page 83
Mathematical operations......Page 86
Rewriting the particle simulator in NumPy......Page 87
Reaching optimal performance with numexpr......Page 91
Pandas fundamentals......Page 93
Indexing Series and DataFrame objects......Page 95
Database-style operations with Pandas......Page 97
Mapping......Page 98
Grouping, aggregations, and transforms......Page 100
Joining......Page 102
Summary......Page 104
Compiling Cython extensions......Page 105
Variables......Page 108
Functions......Page 110
Classes......Page 111
Sharing declarations......Page 113
C arrays and pointers......Page 114
NumPy arrays......Page 117
Typed memoryviews......Page 118
Particle simulator in Cython......Page 120
Profiling Cython......Page 124
Using Cython with Jupyter......Page 128
Summary......Page 131
Chapter 5: Exploring Compilers......Page 133
First steps with Numba......Page 134
Type specializations......Page 136
Object mode versus native mode......Page 137
Universal functions with Numba......Page 140
Generalized universal functions......Page 142
JIT classes......Page 145
Limitations in Numba......Page 148
The PyPy project......Page 149
Setting up PyPy......Page 150
Running a particle simulator in PyPy......Page 151
Other interesting projects......Page 152
Summary......Page 153
Chapter 6: Implementing Concurrency......Page 154
Waiting for I/O......Page 155
Concurrency......Page 156
Callbacks......Page 158
Futures......Page 161
Event loops......Page 163
The asyncio framework......Page 166
Coroutines......Page 167
Converting blocking code into non-blocking code......Page 171
Observables......Page 173
Useful operators......Page 176
Hot and cold observables......Page 180
Building a CPU monitor......Page 183
Summary......Page 186
Chapter 7: Parallel Processing......Page 187
Introduction to parallel programming......Page 188
Graphic processing units......Page 190
Using multiple processes......Page 191
The Process and Pool classes......Page 192
The Executor interface......Page 194
Monte Carlo approximation of pi......Page 195
Synchronization and locks......Page 198
Parallel Cython with OpenMP......Page 201
Automatic parallelism......Page 203
Getting started with Theano......Page 204
Profiling Theano......Page 209
Tensorflow......Page 211
Running code on a GPU......Page 213
Summary......Page 217
Chapter 8: Advanced Introduction to Concurrent and Parallel Programming......Page 218
Concurrent versus sequential......Page 219
Example 1 – checking whether a non-negative number is prime......Page 220
Concurrent versus parallel......Page 223
Not everything should be made concurrent......Page 224
Inherently sequential......Page 225
Example 2 – inherently sequential tasks......Page 226
The history, present, and future of concurrency......Page 228
The history of concurrency......Page 229
The present......Page 230
The future......Page 232
A brief overview of mastering concurrency in Python......Page 234
Why Python?......Page 235
General setup......Page 237
Summary......Page 238
Further reading......Page 239
Technical requirements......Page 240
Terminology......Page 241
The formula for Amdahl's Law......Page 242
Implications......Page 243
Amdahl's Law's relationship to the law of diminishing returns......Page 244
How to simulate in Python......Page 245
Practical applications of Amdahl's Law......Page 249
Summary......Page 250
Further reading......Page 251
Chapter 10: Working with Threads in Python......Page 252
Threads versus processes......Page 253
Multithreading......Page 254
An example in Python......Page 256
The threading module in Python 3......Page 260
Creating a new thread in Python......Page 261
Starting a thread with the thread module......Page 262
Starting a thread with the threading module......Page 264
The concept of thread synchronization......Page 267
An example in Python......Page 268
A connection between real-life and programmatic queues......Page 270
The queue module......Page 271
Queuing in concurrent programming......Page 272
Multithreaded priority queue......Page 276
Summary......Page 277
Further reading......Page 278
Technical requirements......Page 279
Starting from managing files......Page 280
The with statement as a context manager......Page 281
The with statement in concurrent programming......Page 283
Example of deadlock handling......Page 284
Questions......Page 286
Further reading......Page 287
Technical requirements......Page 288
HTML......Page 289
HTTP requests......Page 291
HTTP status code......Page 292
The requests module......Page 293
Making a request in Python......Page 294
Running a ping test......Page 296
Concurrent web requests......Page 297
Spawning multiple threads......Page 298
Refactoring request logic......Page 300
Support from httpstat.us and simulation in Python......Page 302
Timeout specifications......Page 303
Error handling......Page 307
Avoid making a large number of requests......Page 308
Further reading......Page 310
Chapter 13: Working with Processes in Python......Page 311
The concept of a process......Page 312
Processes versus threads......Page 314
Multiprocessing......Page 315
Introductory example in Python......Page 317
The process class......Page 319
The Pool class......Page 320
Determining the current process......Page 321
Waiting for processes......Page 324
Interprocess communication......Page 327
Message passing for a single worker......Page 328
Message passing between several workers......Page 330
Summary......Page 336
Further reading......Page 337
Technical requirements......Page 338
Properties of a reduction operator......Page 339
Examples and non-examples......Page 340
Example implementation in Python......Page 342
Summary......Page 347
Further reading......Page 348
Technical requirements......Page 349
Python as an image processing tool......Page 350
Installing OpenCV and NumPy......Page 351
RGB values......Page 352
Coordinates inside an image......Page 353
OpenCV API......Page 354
Image processing techniques......Page 356
Grayscaling......Page 357
Thresholding......Page 359
Applying concurrency to image processing......Page 364
Choosing the correct way (out of many)......Page 368
Summary......Page 371
Further reading......Page 372
Technical requirements......Page 373
A quick analogy......Page 374
Asynchronous versus other programming models......Page 375
Asynchronous versus synchronous programming......Page 376
Asynchronous versus threading and multiprocessing......Page 377
An example in Python......Page 378
Questions......Page 381
Further reading......Page 382
Technical requirements......Page 383
Coroutines, event loops, and futures......Page 384
Asyncio API......Page 386
The asyncio framework in action......Page 387
Asynchronously counting down......Page 388
A note about blocking functions......Page 392
Asynchronous prime-checking......Page 393
Improvements from Python 3.7......Page 397
Inherently blocking tasks......Page 398
concurrent.futures as a solution for blocking tasks......Page 399
Examples in Python......Page 400
Summary......Page 404
Questions......Page 405
Further reading......Page 406
Chapter 18: Building Communication Channels with asyncio......Page 407
Communication protocol layers......Page 408
Asynchronous programming for communication channels......Page 410
Transports and protocols in asyncio......Page 411
The big picture of asyncio's server client......Page 413
Starting a server......Page 414
Installing Telnet......Page 416
Simulating a connection channel......Page 417
Sending messages back to clients......Page 418
Closing the transports......Page 419
Client-side communication with aiohttp......Page 421
Fetching a website's HTML code......Page 422
Writing files asynchronously......Page 424
Summary......Page 426
Further reading......Page 427
Technical requirements......Page 428
The Dining Philosophers problem......Page 429
Deadlock in a concurrent system......Page 432
Python simulation......Page 433
Implementing ranking among resources......Page 437
Ignoring locks and sharing resources......Page 443
An additional note about locks......Page 445
The concept of livelock......Page 446
Further reading......Page 449
Technical requirements......Page 450
What is starvation?......Page 451
Scheduling......Page 452
Causes of starvation......Page 453
Starvation's relationship to deadlock......Page 454
Problem statement......Page 455
The first readers-writers problem......Page 456
The second readers-writers problem......Page 460
The third readers-writers problem......Page 463
Solutions to starvation......Page 465
Summary......Page 466
Further reading......Page 467
Technical requirements......Page 468
Critical sections......Page 469
How race conditions occur......Page 470
Simulating race conditions in Python......Page 472
The effectiveness of locks......Page 474
Implementation in Python......Page 476
The downside of locks......Page 477
Turning a concurrent program sequential......Page 478
Locks do not lock anything......Page 480
Security......Page 481
Operating systems......Page 482
Networking......Page 483
Questions......Page 484
Further reading......Page 485
Technical requirements......Page 486
An analysis of memory management in Python......Page 487
The problem that the GIL addresses......Page 490
Problems raised by the GIL......Page 491
How to work with the GIL......Page 493
Implementing multiprocessing, rather than multithreading ......Page 494
Summary......Page 496
Further reading......Page 497
Chapter 23: The Factory Pattern......Page 498
The factory method......Page 499
Use cases......Page 500
Implementing the factory method......Page 501
Real-world examples......Page 509
Implementing the abstract factory pattern......Page 510
Summary......Page 515
Chapter 24: The Builder Pattern......Page 516
Real-world examples......Page 517
Use cases......Page 518
Implementation......Page 522
Summary......Page 528
Chapter 25: Other Creational Patterns......Page 529
Real-world examples......Page 530
Implementation......Page 531
Real-world examples......Page 535
Implementation......Page 536
Summary......Page 541
Real-world examples......Page 542
Implementation......Page 543
Summary......Page 546
Chapter 27: The Decorator Pattern......Page 547
Use cases......Page 548
Implementation......Page 549
Summary......Page 554
Real-world examples......Page 555
Implementation......Page 556
Summary......Page 560
Chapter 29: The Facade Pattern......Page 561
Use cases......Page 562
Implementation......Page 563
Summary......Page 567
Chapter 30: Other Structural Patterns......Page 568
The flyweight pattern......Page 569
Use cases......Page 570
Implementation......Page 571
The model-view-controller pattern......Page 576
Real-world examples......Page 577
Use cases......Page 578
Implementation......Page 579
The proxy pattern......Page 583
Use cases......Page 586
Implementation......Page 587
Summary......Page 591
Chapter 31: The Chain of Responsibility Pattern......Page 592
Real-world examples......Page 593
Use cases......Page 595
Implementation......Page 596
Summary......Page 601
Chapter 32: The Command Pattern......Page 602
Use cases......Page 603
Implementation......Page 604
Summary......Page 612
Real-world examples......Page 613
Use cases......Page 614
Implementation......Page 615
Summary......Page 621
Appendix......Page 622
Other Books You May Enjoy......Page 641
Index......Page 644