Hands-On Software Engineering with Python: Move Beyond Basic Programming and Construct Reliable and Efficient Software with Complex Code

✍ Scribed by Allbee, Brian

• Publisher: Packt Publishing Ltd
• Year: 2018
• Tongue: English
• Leaves: 723
• Category: Library

✦ Synopsis

Learn how to apply real-world Software Engineering disciplines by seeing them applied to a Python system, from fundamentals through high-end systems.Key Features Learn what skills, technical and otherwise, are expected from Senior Developers Understand how the skills are applied to a realistically-scoped project Release the power of general and Python development best practices Book DescriptionSoftware Engineering is about more than just writing code — it includes a host of “soft” skills that apply to almost any development-effort, no matter the language, development-methodology or scope of the project. Being a Senior Developer all but requires awareness of how those skills, along with their expected technical counterparts, mesh together through a project’s life-cycle.This book walks you through that discovery by walking through the entire life-cycle of a multi-tier system and its related software projects. You’ll see what happens before any development takes place, and what impact the decisions and designs made at each step have on the development process. The development of the entire project, over the course of several iterations based on real-world Agile iterations, will be implemented, sometimes starting with nothing, in one of the fastest-growing languages in the world — Python. The details of, and advantages behind a number industry best practices will show why they are best practices. Application of those practices in Python will be laid out, and some Python-specific capabilities that are often overlooked. Finally, the book will, from first principles to complete implementation, implement a High Performance Computing solution, and show the application of Metaprogramming concepts to the book’s project.What you will learn Learn what happens over the course of a system’s life (SDLC). Learn what to expect from the pre-development life-cycle steps. Find out how the development-specific phases of the SDLC affect development. Uncover what a real-world development process might be like, in an Agile way. Find out how to do more than just write the code. Learn project-independent best practices exist, and how to use them. Find out how to design and implement a High Performance Computing process. Learn what Metaprogramming is, can do, and what risks come with it Who This Book Is ForThis book is for existing developers who have basic understanding of Programming and Programming paradigms and would like to skill up as a Senior Programmer. Basic Python knowledge is assumedAbout the AuthorBrian Allbeehas been a developer professionally for two decades. His experience includes web and system applications in several languages, for industries as varied as organization membership services, consumer health-care reporting, advertising, and cloud automation. He fell in love with Python just before 2.6 was available, and has focused his development efforts as much on Python as time and professional obligations allowed since then. He lives and codes in the Denver area.

✦ Table of Contents

Cover......Page 1
Title Page......Page 2
Copyright and Credits......Page 3
Packt Upsell......Page 4
Contributors......Page 5
Table of Contents......Page 7
Preface......Page 14
Chapter 1: Programming versus Software Engineering......Page 21
The bigger picture......Page 22
Asking questions......Page 23
Summary......Page 31
Chapter 2: The Software Development Life Cycle......Page 32
Pre-development phases of the SDLC......Page 33
Initial concept/vision......Page 34
Concept development......Page 36
Project management planning......Page 38
System architecture and design......Page 40
System integration, testing, and acceptance......Page 41
Post-development phases of the SDLC......Page 42
Summary......Page 44
Architecture, both logical and physical......Page 45
Logical architecture......Page 46
Physical architecture......Page 48
Use cases (business processes and rules)......Page 50
Data structure and flow......Page 52
Interprocess communication......Page 54
System scope and scale......Page 57
Summary......Page 59
Chapter 4: Methodologies, Paradigms, and Practices......Page 60
Process methodologies......Page 61
Waterfall......Page 62
Agile (in general)......Page 66
Scrum......Page 69
Scrum and the phases of the SDLC model......Page 76
Kanban......Page 77
Kanban and the phases of the SDLC model......Page 79
Extreme programming......Page 80
Feature-driven development......Page 81
Test-driven design......Page 82
Development paradigms......Page 83
Object-oriented programming......Page 84
Functional programming......Page 87
Development practices......Page 89
Continuous integration......Page 90
Continuous delivery or deployment......Page 91
Summary......Page 92
Goals for the system......Page 93
What's known/designed before development starts......Page 95
What the iteration chapters will look like......Page 98
Iteration goals and stories......Page 99
Post-development considerations and impact......Page 100
Summary......Page 101
Development tools......Page 102
Integrated Development Environment (IDE) options......Page 103
IDLE......Page 105
Geany......Page 107
Eclipse variations + PyDev......Page 109
Others......Page 112
Source Code Management......Page 113
Typical SCM activities......Page 115
Git......Page 117
Basic workflows for Git and SVN compared......Page 118
Best practices......Page 120
Standards for code......Page 121
PEP-8......Page 122
Code organization in modules......Page 125
Structure and standards for classes......Page 129
Function and method annotation (hinting)......Page 135
Process standards......Page 138
Unit testing......Page 139
Repeatable build processes......Page 145
Integrating unit tests and build processes......Page 150
Defining package structures for Python code......Page 151
Packages in a project's context......Page 153
Using Python virtual environments......Page 154
Summary......Page 158
Chapter 7: Setting Up Projects and Processes......Page 160
Assembly of stories and tasks......Page 161
Setting Up SCM......Page 162
Component project analysis......Page 169
Component project setup......Page 170
Packaging and build process......Page 171
Python virtual environments......Page 176
Basic unit testing......Page 184
Identifying missing test case classes......Page 186
Identifying missing test methods......Page 190
Creating reusable module code coverage tests......Page 195
The property and method testing decorators......Page 203
Creating unit test template files......Page 206
Integrating tests with the build process......Page 211
Summary......Page 213
Chapter 8: Creating Business Objects......Page 214
Iteration goals......Page 215
Assembly of stories and tasks......Page 216
A quick review of classes......Page 218
Address......Page 219
BaseArtisan......Page 229
OO principles – composition over inheritance......Page 231
Implementing BaseArtisan's properties......Page 234
Implementing BaseArtisan's methods......Page 244
BaseCustomer......Page 249
BaseOrder......Page 250
BaseProduct......Page 252
Dealing with duplicated code – HasProducts......Page 259
Summary......Page 266
Chapter 9: Testing Business Objects......Page 267
Starting the unit testing process......Page 268
Unit testing the Address class......Page 274
Unit testing HasProducts......Page 287
Unit testing BaseProduct......Page 290
Unit testing BaseOrder......Page 294
Unit-testing BaseCustomer......Page 296
Unit testing BaseArtisan......Page 298
Unit testing patterns established so far......Page 300
Distribution and installation considerations......Page 301
Quality assurance and acceptance......Page 304
Operation/use, maintenance, and decommissioning considerations......Page 305
Summary......Page 306
Chapter 10: Thinking About Business Object Data Persistence......Page 307
Iterations are (somewhat) flexible......Page 308
Data storage options......Page 310
Relational databases......Page 311
Advantages and drawbacks......Page 312
MySQL/MariaDB......Page 313
NoSQL databases......Page 314
Advantages and drawbacks......Page 317
Other NoSQL options......Page 318
Other data storage options......Page 319
Selecting a data storage option......Page 320
Polymorphism (and programming to an interface)......Page 322
Data access design strategies......Page 323
Why start from scratch?......Page 330
Summary......Page 331
Chapter 11: Data Persistence and BaseDataObject......Page 332
The BaseDataObject ABC......Page 333
Unit testing BaseDataObject......Page 350
Summary......Page 359
Chapter 12: Persisting Object Data to Files......Page 360
Setting up the hms_artisan project......Page 361
Creating a local file system data store......Page 362
Implementing JSONFileDataObject......Page 364
The concrete business objects of hms_artisan......Page 378
Dealing with is_dirty and properties......Page 379
hms_artisan.Artisan......Page 381
hms_artisan.Product......Page 387
hms_artisan.Order......Page 390
Summary......Page 402
Chapter 13: Persisting Data to a Database......Page 403
The Artisan Gateway and Central Office application objects......Page 404
Picking out a backend datastore engine......Page 405
The data access strategy for the Central Office projects......Page 410
Supporting objects for data persistence......Page 411
RDBMS implementations......Page 431
The concrete business objects of the Central Office projects......Page 439
hms_core.co_objects.Artisan......Page 440
hms_core.co_objects.Product......Page 447
Other hms_core.co_objects classes......Page 453
Accounting for the other CRUD operations......Page 454
Summary......Page 456
Chapter 14: Testing Data Persistence......Page 457
Testing hms_artisan.data_storage......Page 458
Testing hms_artisan.artisan_objects......Page 468
Testing the new hms_core Classes......Page 484
Unit testing hms_core.data_storage.py......Page 485
Building/distribution, demonstration, and acceptance......Page 494
Summary......Page 498
Chapter 15: Anatomy of a Service......Page 499
Service structure......Page 500
Configuration......Page 504
Windows-style .ini files......Page 505
JSON files......Page 507
YAML files......Page 508
Logging service activities......Page 509
Handling requests and generating responses......Page 512
Filesystem – based......Page 513
HTTP- or web-based......Page 514
Message- queue-based......Page 516
Other request types......Page 517
Request and response formats......Page 518
A generic service design......Page 519
The BaseDaemon ABC......Page 521
The BaseRequestHandler and BaseResponseFormatter ABCs......Page 536
Running a service using systemctl (Linux)......Page 544
Running a service using NSSM (Windows)......Page 546
macOS, launchd, and launchctl......Page 548
Managing services on other systems......Page 549
Summary......Page 550
Chapter 16: The Artisan Gateway Service......Page 551
Overview and goal......Page 552
Iteration stories......Page 555
Messages......Page 556
Deciding on a message-transmission mechanism......Page 566
Message-queue implementation with RabbitMQ......Page 568
Handling messages......Page 580
Queues and related Artisan properties......Page 585
Requirements for a web-service-based daemon......Page 591
Traffic to and from the service......Page 593
Impacts on testing and deployment......Page 598
Summary......Page 599
Chapter 17: Handling Service Transactions......Page 600
Remaining stories......Page 601
A bit of reorganization......Page 603
Preparation for object transactions......Page 604
Product object transactions......Page 606
Artisan – creating a product......Page 607
Central Office – approving/listing a product......Page 609
Central Office – altering product data......Page 611
Artisan – updating product data......Page 613
Artisan object transactions......Page 614
Central Office – creating an artisan......Page 617
Central Office – updating artisan data......Page 618
Artisan – updating Artisan data......Page 619
Order object transactions......Page 620
Customer – relaying order items to artisans......Page 621
Customer – canceling an order......Page 631
Artisan – fulfilling an item in an order......Page 633
When do messages get sent?......Page 635
Summary......Page 643
The challenges of testing services......Page 644
The overall testing strategy......Page 647
Unit testing variations of note......Page 648
Testing Artisan transactions......Page 653
Demonstrating the service......Page 668
Packaging and deploying the service......Page 673
Common considerations across all operating systems......Page 675
Linux (systemd) execution......Page 677
Windows (NSSM) execution......Page 678
Code review, refactoring, and cleanup......Page 681
Developing a UI......Page 683
Summary......Page 685
Chapter 19: Multiprocessing and HPC in Python......Page 686
Common factors to consider......Page 687
A simple but expensive algorithm......Page 689
Some testing setup......Page 690
Local parallel processing......Page 691
Threads......Page 700
Parallelizing across multiple machines......Page 701
Common functionality......Page 703
The Worker nodes......Page 705
The Orchestrator......Page 707
The Dispatcher......Page 708
Integrating Python with large-scale, cluster computing frameworks......Page 709
Python, Hadoop, and Spark......Page 712
Summary ......Page 713
Other Books You May Enjoy......Page 714
Index......Page 717

✦ Subjects

Nonfiction