Preface to the Second Edition
Preface to the First Edition
What You Need
Conventions
Example Code and Sample Solutions
Contents
1 Introduction
1.1 Introduction
1.2 Useful Python Resources
Part I Advanced Language Features
2 Python Type Hints
2.1 Introduction
2.2 Pythons Type System
2.3 The Challenge for Python Developers
2.4 Static Typing
2.5 Python Type Hints
2.6 Type Hint Layout
2.7 Type Hints for Multiple Types
2.8 The Self Type
2.9 The Benefits of Type Hints
2.10 Summary
2.11 Online Resources
3 Class Slots
3.1 Introduction
3.2 The Problem
3.3 Slots to the Rescue
3.4 Performance Benefits
3.5 Why Not Use Slots?
3.6 Online Resource
4 Weak References
4.1 Introduction
4.2 How Garbage Collection Works: Reference Counting
4.3 Weak References
4.4 When to Use Weak References
4.5 The Weakref Module
4.6 Creating Weak References
4.7 Retrieving Objects from Weak References
4.8 The WeakValueDicitonary
4.9 WeakKeyDictionary
4.10 Proxy Objects
4.11 Online Resources
5 Data Classes
5.1 Introduction
5.2 A Traditional Class
5.3 Defining Data Classes
5.4 Defining Additional Behaviour
5.5 The Dataclass Decorator
5.6 Custom Factory for Fields
5.7 Immutable Dataclasses
5.8 Data Classes and Inheritance
5.9 Post Initialisation
5.10 Initialisation Variables
5.11 Positional Attributes
5.12 Exercise
6 Structural Pattern Matching
6.1 Introduction
6.2 The Match Statement
6.3 Matching Classes with Positional Attributes
6.4 Matching Against Standard Classes
6.5 Online Resource
7 Working with pprint
7.1 Introduction
7.2 The pprint Data Printer Module
7.3 Basic pprint Usage
7.4 Changing the Width
7.5 Changing the Depth
7.6 Managing the Indentation Level
7.7 Reducing Line Breaks Using Compact
7.8 The pformat Function
7.9 The saferepr() Function
7.10 Using the PrettyPrinter Class
7.11 Online Resource
8 Shallow v Deep Copy
8.1 Introduction
8.2 Copying a List of Lists
8.3 The Problem with Copying
8.4 The Copy Module to the Rescue
8.5 Using the deepcopy() Function
8.6 Online Resource
9 The init Versus new and call
9.1 Introduction
9.2 The __new and init Methods
9.3 The new Method
9.4 When to Use the new Method
9.5 Using new to Create a Singleton Object
9.6 The init Method
9.7 Can new and init Be Used Together?
9.8 The call Method
9.9 Summary
10 Python Metaclasses and Meta Programming
10.1 Introduction
10.2 Metaprogramming
10.3 Decorators as a Form of Metaprogramming
10.4 Metaclasses for Metaprogramming
10.4.1 Singleton Metaclass
10.5 Exec and Eval for Metaprogramming
10.5.1 The exec() Function
10.5.2 The eval() Function
10.5.3 eval Versus exec()
Part II Computer Graphics and GUIs
11 Introduction to Computer Graphics
11.1 Introduction
11.2 Background
11.3 The Graphical Computer Era
11.4 Interactive and Non Interactive Graphics
11.5 Pixels
11.6 Bit Map Versus Vector Graphics
11.7 Buffering
11.8 Python and Computer Graphics
11.9 References
11.10 Online Resources
12 Python Turtle Graphics
12.1 Introduction
12.2 The Turtle Graphics Library
12.2.1 The Turtle Module
12.2.2 Basic Turtle Graphics
12.2.3 Drawing Shapes
12.2.4 Filling Shapes
12.3 Other Graphics Libraries
12.4 3D Graphics
12.4.1 PyOpenGL
12.5 Online Resources
12.6 Exercises
13 Computer Generated Art
13.1 Creating Computer Art
13.2 A Computer Art Generator
13.3 Fractals in Python
13.4 The Koch Snowflake
13.5 Mandelbrot Set
13.6 Online Resources
13.7 Exercises
14 Introduction to Matplotlib
14.1 Introduction
14.2 Matplotlib
14.3 Plot Components
14.4 Matplotlib Architecture
14.4.1 Backend Layer
14.4.2 The Artist Layer
14.4.3 The Scripting Layer
14.5 Online Resources
15 Graphing with Matplotlib Pyplot
15.1 Introduction
15.2 The pyplot API
15.3 Line Graphs
15.3.1 Coded Format Strings
15.4 Scatter Graph
15.4.1 When to Use Scatter Graphs
15.5 Pie Charts
15.5.1 Expanding Segments
15.5.2 When to Use Pie Charts
15.6 Bar Charts
15.6.1 Horizontal Bar Charts
15.6.2 Coloured Bars
15.6.3 Stacked Bar Charts
15.6.4 Grouped Bar Charts
15.7 Figures and Subplots
15.8 3D Graphs
15.9 Exercises
16 Graphical User Interfaces
16.1 Introduction
16.2 GUIs and WIMPS
16.3 Windowing Frameworks for Python
16.3.1 Platform-Independent GUI Libraries
16.3.2 Platform-Specific GUI Libraries
16.4 Online Resources
17 Tkinter GUI Library
17.1 Introduction
17.2 Tkinter
17.3 Windows as Objects
17.4 Key Concepts
17.4.1 The Tk Class
17.4.2 TK Widgets
17.4.3 The TopLevel Class
17.4.4 The Frame Class
17.4.5 Dialogs
17.4.6 The Canvas Class
17.5 The Class Inheritance Hierarchy
17.5.1 Layout Management
17.6 A Simple Example
17.7 Tkinter Installation
17.7.1 Mac Installation
17.7.2 Windows Installation
17.8 GUI Builders for Tkinter
17.9 Online Resources
17.10 Exercises
18 Events in Tkinter User Interfaces
18.1 Introduction
18.2 Event Handling
18.3 What is Event Handling?
18.4 What Are Event Handlers?
18.5 Event Binders
18.6 Virtual Events
18.7 Event Definitions
18.8 What Types of Event Are There?
18.9 Binding an Event to an Event Handler
18.10 Implementing Event Handling
18.11 An Interactive GUI Application
18.12 Online Resources
18.13 Exercises
19 PyDraw Tkinter Example Application
19.1 Introduction
19.2 The PyDraw Application
19.3 The Structure of the Application
19.3.1 Model, View and Controller Architecture
19.3.2 PyDraw MVC Architecture
19.3.3 Additional Classes
19.3.4 Object Relationships
19.4 The Interactions Between Objects
19.4.1 The PyDrawApp
19.5 The PyDrawView Constructor
19.5.1 Changing the Application Mode
19.5.2 Adding a Graphic Object
19.6 The Classes
19.6.1 The PyDrawConstants Class
19.6.2 The PyDrawView Class
19.6.3 The PyDrawMenuBar Class
19.6.4 The PyDrawController Class
19.6.5 The DrawingModel Class
19.6.6 The DrawingView Class
19.6.7 The DrawingController Class
19.6.8 The Figure Class
19.6.9 The Square Class
19.6.10 The Circle Class
19.6.11 The Line Class
19.6.12 The Text Class
19.7 Reference
19.8 Exercises
Part III Computer Games
20 Introduction to Games Programming
20.1 Introduction
20.2 Games Frameworks and Libraries
20.3 Python Games Development
20.4 Using Pygame
20.5 Online Resources
21 Building Games with Pygame
21.1 Introduction
21.2 The Display Surface
21.3 Events
21.3.1 Event Types
21.3.2 Event Information
21.3.3 The Event Queue
21.4 A First pygame Application
21.5 Further Concepts
21.6 A More Interactive pygame Application
21.7 Alternative Approach to Processing Input Devices
21.8 pygame Modules
21.9 Online Resources
22 StarshipMeteors Pygame
22.1 Introduction
22.2 Creating a Spaceship Game
22.3 The Main Game Class
22.4 The GameObject Class
22.5 Displaying the Starship
22.6 Moving the Spaceship
22.7 Adding a Meteor Class
22.8 Moving the Meteors
22.9 Identifying a Collision
22.10 Identifying a Win
22.11 Increasing the Number of Meteors
22.12 Pausing the Game
22.13 Displaying the Game Over Message
22.14 The StarshipMeteors Game
22.15 Online Resources
22.16 Exercises
Part IV Testing
23 Introduction to Testing
23.1 Introduction
23.2 Types of Testing
23.3 What Should Be Tested?
23.4 Types of Testing
23.4.1 Unit Testing
23.4.2 Integration Testing
23.4.3 System Testing
23.4.4 Installation/Upgrade Testing
23.4.5 Smoke Tests
23.5 Automating Testing
23.6 Test-Driven Development
23.6.1 The TDD Cycle
23.6.2 Test Complexity
23.6.3 Refactoring
23.7 Design for Testability
23.7.1 Testability Rules of Thumb
23.8 Online Resources
23.9 Book Resources
24 PyTest Testing Framework
24.1 Introduction
24.2 What is PyTest?
24.3 Setting up PyTest
24.4 A Simple PyTest Example
24.5 Working with PyTest
24.6 Parameterised Tests
24.7 Online Resources
24.8 Exercises
25 Mocking for Testing
25.1 Introduction
25.2 Why Mock?
25.3 What is Mocking?
25.4 Common Mocking Framework Concepts
25.5 Mocking Frameworks for Python
25.6 The Unittest.Mock Library
25.6.1 Mock and Magic Mock Classes
25.6.2 The Patchers
25.6.3 Mocking Returned Objects
25.6.4 Validating Mocks Have Been Called
25.7 Mock and MagicMock Usage
25.7.1 Naming Your Mocks
25.7.2 Mock Classes
25.7.3 Attributes on Mock Classes
25.7.4 Mocking Constants
25.7.5 Mocking Properties
25.7.6 Raising Exceptions with Mocks
25.7.7 Applying Patch to Every Test Method
25.7.8 Using Patch as a Context Manager
25.8 Mock Where You Use It
25.9 Patch Order Issues
25.10 How Many Mocks?
25.11 Mocking Considerations
25.12 Online Resources
25.13 Exercises
Part V File Input/Output
26 Introduction to Files, Paths and IO
26.1 Introduction
26.2 File Attributes
26.3 Paths
26.4 File Input/Output
26.5 Sequential Access versus Random Access
26.6 Files and I/O in Python
26.7 Online Resources
27 Reading and Writing Files
27.1 Introduction
27.2 Obtaining References to Files
27.3 Reading Files
27.4 File Contents Iteration
27.5 Writing Data to Files
27.6 Using Files and with Statements
27.7 The Fileinput Module
27.8 Renaming Files
27.9 Deleting Files
27.10 Random Access Files
27.11 Directories
27.12 Temporary Files
27.13 Working with Paths
27.14 Online Resources
27.15 Exercise
28 Stream IO
28.1 Introduction
28.2 What is a Stream?
28.3 Python Streams
28.4 IOBase
28.5 Raw IO/UnBuffered IO Classes
28.6 Binary IO/Buffered IO Classes
28.7 Text Stream Classes
28.8 Stream Properties
28.9 Closing Streams
28.10 Returning to the Open() Function
28.11 Online Resource
28.12 Exercise
29 Working with CSV Files
29.1 Introduction
29.2 CSV Files
29.2.1 The CSV Writer Class
29.2.2 The CSV Reader Class
29.2.3 The CSV DictWriter Class
29.2.4 The CSV DictReader Class
29.3 Online Resources
29.4 Exercises
30 Working with Excel Files
30.1 Introduction
30.2 Excel Files
30.3 The Openpyxl. Workbook Class
30.4 The Openpyxl. WorkSheet Objects
30.5 Working with Cells
30.6 Sample Excel File Creation Application
30.7 Loading a Workbook from an Excel File
30.8 Online Resources
30.9 Exercises
31 Regular Expressions in Python
31.1 Introduction
31.2 What Are Regular Expressions?
31.3 Regular Expression Patterns
31.3.1 Pattern Metacharacters
31.3.2 Special Sequences
31.3.3 Sets
31.4 The Python re Module
31.5 Working with Python Regular Expressions
31.5.1 Using Raw Strings
31.5.2 Simple Example
31.5.3 The Match Object
31.5.4 The search() Function
31.5.5 The match() Function
31.5.6 The Difference Between Matching and Searching
31.5.7 The finadall() Function
31.5.8 The finditer() Function
31.5.9 The split() Function
31.5.10 The sub() Function
31.5.11 The compile() Function
31.6 Online Resources
31.7 Exercises
Part VI Database Access
32 Introduction to Databases
32.1 Introduction
32.2 What Is a Database?
32.2.1 Data Relationships
32.2.2 The Database Schema
32.3 SQL and Databases
32.4 Data Manipulation Language
32.5 Transactions in Databases
32.6 Further Reading
33 Python DB-API
33.1 Accessing a Database from Python
33.2 The DB-API
33.2.1 The Connect Function
33.2.2 The Connection Object
33.2.3 The Cursor Object
33.2.4 Mappings from Database Types to Python Types
33.2.5 Generating Errors
33.2.6 Row Descriptions
33.3 Transactions in PyMySQL
33.4 Online Resources
34 PyMySQL Module
34.1 The PyMySQL Module
34.2 Working with the PyMySQL Module
34.2.1 Importing the Module
34.2.2 Connect to the Database
34.2.3 Obtaining the Cursor Object
34.2.4 Using the Cursor Object
34.2.5 Obtaining Information About the Results
34.2.6 Fetching Results
34.2.7 Close the Connection
34.3 Complete PyMySQL Query Example
34.4 Inserting Data to the Database
34.5 Updating Data in the Database
34.6 Deleting Data in the Database
34.7 Creating Tables
34.8 Online Resources
34.9 Exercises
Part VII Logging
35 Introduction to Logging
35.1 Introduction
35.2 Why Log?
35.3 What is the Purpose of Logging?
35.4 What Should You Log?
35.5 What not to Log
35.6 Why not Just Use Print?
35.7 Online Resources
36 Logging in Python
36.1 The Logging Module
36.2 The Logger
36.3 Controlling the Amount of Information Logged
36.4 Logger Methods
36.5 Default Logger
36.6 Module Level Loggers
36.7 Logger Hierarchy
36.8 Formatters
36.8.1 Formatting Log Messages
36.8.2 Formatting Log Output
36.9 Online Resources
36.10 Exercises
37 Advanced Logging
37.1 Introduction
37.2 Handlers
37.2.1 Setting the Root Output Handler
37.2.2 Programmatically Setting the Handler
37.2.3 Multiple Handlers
37.3 Filters
37.4 Logger Configuration
37.5 Performance Considerations
37.6 Exercises
Part VIII Concurrency and Parallelism
38 Introduction to Concurrency and Parallelism
38.1 Introduction
38.2 Concurrency
38.3 Parallelism
38.4 Distribution
38.5 Grid Computing
38.6 Concurrency and Synchronisation
38.7 Object Orientation and Concurrency
38.8 Threads V Processes
38.9 Some Terminology
38.10 Online Resources
39 Threading
39.1 Introduction
39.2 Threads
39.2.1 Thread States
39.2.2 Creating a Thread
39.2.3 Instantiating the Thread Class
39.3 The Thread Class
39.4 The Threading Module Functions
39.5 Passing Arguments to a Thread
39.6 Extending the Thread Class
39.7 Daemon Threads
39.8 Naming Threads
39.9 Thread Local Data
39.10 Timers
39.11 The Global Interpreter Lock
39.12 Online Resources
39.13 Exercise
40 MultiProcessing
40.1 Introduction
40.2 The Process Class
40.3 Working with the Process Class
40.4 Alternative Ways to Start a Process
40.5 Using a Pool
40.6 Exchanging Data Between Processes
40.7 Sharing State Between Processes
40.7.1 Process Shared Memory
40.8 Online Resources
40.9 Exercises
41 Inter Thread/Process Synchronisation
41.1 Introduction
41.2 Using a Barrier
41.3 Event Signalling
41.4 Synchronising Concurrent Code
41.5 Python Locks
41.6 Python Conditions
41.7 Python Semaphores
41.8 The Concurrent Queue Class
41.9 Online Resources
41.10 Exercises
42 Futures
42.1 Introduction
42.2 The Need for a Future
42.3 Futures in Python
42.3.1 Future Creation
42.3.2 Simple Example Future
42.4 Running Multiple Futures
42.4.1 Waiting for All Futures to Complete
42.4.2 Processing Results as Completed
42.5 Processing Future Results Using a Callback
42.6 Online Resources
42.7 Exercises
43 Concurrency with AsyncIO
43.1 Introduction
43.2 Asynchronous IO
43.3 Async IO Event Loop
43.4 The Async and Await Keywords
43.4.1 Using Async and Await
43.5 Async IO Tasks
43.6 Running Multiple Tasks
43.6.1 Collating Results from Multiple Tasks
43.6.2 Handling Task Results as They Are Made Available
43.7 Online Resources
43.8 Exercises
44 Performance Monitoring and Profiling
44.1 Introduction
44.2 Why Monitor Performance and Memory?
44.3 Performance Monitoring and Profiling
44.4 Performance Monitoring
44.4.1 The Time Module
44.4.2 The Timeit Module
44.4.3 The Psutil Module
44.5 Python Profiling
44.5.1 The cProfile Module
44.5.2 The Line_Profiler Module
44.5.3 The Memory_Profiler Module
44.5.4 Additional Third-Party Libraries
44.6 Profiling with cProfile
44.7 Memory Profiling
44.8 Online Resources
Part IX Reactive Programming
45 Reactive Programming Introduction
45.1 Introduction
45.2 What Is a Reactive Application?
45.3 The ReactiveX Project
45.4 The Observer Pattern
45.5 Hot and Cold Observables
45.6 Differences Between Event Driven Programming and Reactive Programming
45.7 Advantages of Reactive Programming
45.8 Disadvantages of Reactive Programming
45.9 The RxPy Reactive Programming Framework
45.10 Online Resources
46 RxPy Observables, Observers and Subjects
46.1 Introduction
46.2 RxPy Library
46.3 Observables in RxPy
46.4 Observers in RxPy
46.5 Multiple Subscribers/Observers
46.6 Subjects in RxPy
46.7 Observer Concurrency
46.7.1 Available Schedulers
46.8 Online Resources
46.9 Exercises
47 RxPy Operators
47.1 Introduction
47.2 Reactive Programming Operators
47.3 Piping Operators
47.4 Creational Operators
47.5 Transformational Operators
47.6 Combinatorial Operators
47.7 Filtering Operators
47.8 Mathematical Operators
47.9 Chaining Operators
47.10 Online Resources
47.11 Exercises
Part X Network Programming
48 Introduction to Sockets and Web Services
48.1 Introduction
48.2 Sockets
48.3 Web Services
48.4 Addressing Services
48.5 Localhost
48.6 Port Numbers
48.7 IPv4 Versus IPv6
48.8 Sockets and Web Services in Python
48.9 Online Resources
49 Sockets in Python
49.1 Introduction
49.2 Socket to Socket Communication
49.3 Setting up a Connection
49.4 An Example Client Server Application
49.4.1 The System Structure
49.4.2 Implementing the Server Application
49.4.3 Socket Types and Domains
49.4.4 Implementing the Client Application
49.5 The Socketserver Module
49.6 Http Server
49.7 Online Resources
49.8 Exercises
50 Web Services in Python
50.1 Introduction
50.2 RESTful Services
50.3 A RESTful API
50.4 Python Web Frameworks
50.5 Online Resources
51 Flask Web Services
51.1 Introduction
51.2 Flask
51.3 Hello World in Flask
51.3.1 Using JSON
51.4 Implementing a Flask Web Service
51.4.1 A Simple Service
51.4.2 Providing Routing Information
51.5 Running the Service
51.6 Invoking the RESTFul Service
51.6.1 The Final Solution
51.7 Online Resources
52 Flask Bookshop Web Service
52.1 Introduction
52.2 Building a Flask Bookshop Service
52.3 The Design
52.4 The Domain Model
52.5 Encoding Books into JSON
52.6 Setting Up the GET Services
52.7 Deleting a Book
52.8 Adding a New Book
52.9 Updating a Book
52.10 What Happens if We Get It Wrong?
52.11 Bookshop Services Listing
52.12 Exercises
Part XI Data Science: Data Analytics and Machine Learning
53 Introduction to Data Science
53.1 Introduction
53.2 Data Science
53.3 Data Science Tools and Techniques
53.4 Data Analytics Process
53.5 Python and Data Science
53.6 Machine Learning for Data Science
53.7 Online Resources
54 Pandas and Data Analytics
54.1 Introduction
54.2 The Data
54.2.1 The UK Government COVID Data Set
54.2.2 The Google Mobility Data Set
54.3 Python Pandas
54.3.1 Pandas Series and DataFrames
54.4 Loading and Analysing UK COVID Data Set
54.5 Loading the Google Mobility Data Set
54.6 Merging Two DataFrames
54.7 Analysing the Combined Data
54.8 Summary
55 Alternatives to Pandas
55.1 Introduction
55.2 Comparing Pandas 2.0.0
55.3 Pandas 1.x v 2.x
55.4 Pandas Versus Other Libraries and Tools
55.5 Online Resources
56 Machine Learning in Python
56.1 Introduction
56.2 The Data
56.3 SciKitLearn
56.4 The Problem
56.5 Using Regression Supervised Learning Systems
56.6 K-Nearest Neighbour Regressor
56.7 Decision Tree Regressor
56.8 Random Forest Regressor
56.9 Summary of Metrics Obtained
56.10 Creating the Regressor Object
56.11 Online Resources
57 Pip and Conda Virtual Environments
57.1 Introduction
57.2 Virtual Environments
57.3 Working with Pip
57.3.1 Activating a Pip Environment
57.3.2 Installing Modules Using Pip
57.3.3 Deactivating a Pip Environment
57.3.4 Check Version of Pip
57.3.5 Installing Modules into a Pip Environment
57.3.6 Freezing Modules
57.4 Conda
57.5 Anaconda
57.5.1 Installing Anaconda
57.6 Working with Anaconda
57.6.1 Checking the Conda Version
57.6.2 Updating Conda
57.6.3 Creating a Conda Environment
57.6.4 Listing Available Conda Environments
57.6.5 Activating a Conda Environment
57.6.6 Deactivating a Conda Environment
57.6.7 Listing the Modules Loaded into a Conda Environment
57.6.8 Removing an Anaconda Environment
57.6.9 Installing a Module into a Conda Environment
57.7 Anaconda in PyCharm
57.8 Online Resources