Verification, Validation and Testing of Engineered Systems (Wiley Series in Systems Engineering and Management)

✍ Scribed by A. Engel

• Publisher: Wiley
• Year: 2010
• Tongue: English
• Leaves: 713
• Edition: 1
• Category: Library

✦ Synopsis

Systems' Verification Validation and Testing (VVT) are carried out throughout systems' lifetimes. Notably, quality-cost expended on performing VVT activities and correcting system defects consumes about half of the overall engineering cost. Verification, Validation and Testing of Engineered Systems provides a comprehensive compendium of VVT activities and corresponding VVT methods for implementation throughout the entire lifecycle of an engineered system. In addition, the book strives to alleviate the fundamental testing conundrum, namely: What should be tested? How should one test? When should one test? And, when should one stop testing? In other words, how should one select a VVT strategy and how it be optimized?The book is organized in three parts: The first part provides introductory material about systems and VVT concepts. This part presents a comprehensive explanation of the role of VVT in the process of engineered systems (Chapter-1). The second part describes 40 systems' development VVT activities (Chapter-2) and 27 systems' post-development activities (Chapter-3). Corresponding to these activities, this part also describes 17 non-testing systems' VVT methods (Chapter-4) and 33 testing systems' methods (Chapter-5). The third part of the book describes ways to model systems’ quality cost, time and risk (Chapter-6), as well as ways to acquire quality data and optimize the VVT strategy in the face of funding, time and other resource limitations as well as different business objectives (Chapter-7). Finally, this part describes the methodology used to validate the quality model along with a case study describing a system’s quality improvements (Chapter-8).Fundamentally, this book is written with two categories of audience in mind. The first category is composed of VVT practitioners, including Systems, Test, Production and Maintenance engineers as well as first and second line managers. The second category is composed of students and faculties of Systems, Electrical, Aerospace, Mechanical and Industrial Engineering schools. This book may be fully covered in two to three graduate level semesters; although parts of the book may be covered in one semester. University instructors will most likely use the book to provide engineering students with knowledge about VVT, as well as to give students an introduction to formal modeling and optimization of VVT strategy.

✦ Table of Contents

VERIFICATION, VALIDATION, AND TESTING OF ENGINEERED SYSTEMS......Page 1
Contents......Page 9
Preface......Page 19
Part I: Introduction......Page 23
1.1 OPENING......Page 25
1.1.1 Background......Page 26
1.1.3 Intended Audience......Page 27
1.1.4 Book Structure and Contents......Page 28
1.1.5 Scope of Application......Page 30
1.2.1 Introduction—VVT Systems and Process......Page 31
1.2.2 Engineered Systems......Page 32
1.2.3 VVT Concepts and Definition......Page 34
1.2.4 The Fundamental VVT Dilemma......Page 41
1.2.5 Modeling Systems and VVT Lifecycle......Page 42
1.2.6 Modeling VVT and Risks as Cost and Time Drivers......Page 46
1.3.1 Introduction—Canonical Systems VVT Paradigm......Page 54
1.3.2 Phases of the System Lifecycle......Page 56
1.3.3 Views of the System......Page 59
1.4.1 Introduction......Page 61
1.4.2 VVT Methodology Overview......Page 62
1.4.3 VVT Tailoring......Page 65
1.4.4 VVT Documents......Page 72
1.5 REFERENCES......Page 78
Part II: VVT Activities and Methods......Page 83
2.1.1 Systems Development Lifecycle Phases and VVT Activities......Page 85
2.1.2 VVT Activity Aspects......Page 86
2.2.1 Generate Requirements Verification Matrix (RVM)......Page 87
2.2.2 Generate VVT Management Plan (VVT-MP)......Page 89
2.2.3 Assess the Request For Proposal (RFP) Document......Page 91
2.2.4 Assess System Requirements Specification (SysRS)......Page 93
2.2.5 Assess Project Risk Management Plan (RMP)......Page 94
2.2.6 Assess System Safety Program Plan (SSPP)......Page 96
2.2.8 Participate in System Engineering Management Plan (SEMP) Review......Page 99
2.2.9 Conduct Engineering Peer Review of the VVT-MP Document......Page 101
2.3.1 Optimize the VVT Strategy......Page 102
2.3.2 Assess System/Subsystem Design Description (SSDD)......Page 105
2.3.3 Validate System Design by Means of Virtual Prototype......Page 107
2.3.4 Validate System Design Tools......Page 108
2.3.5 Assess System Design for Meeting Future Lifecycle Needs......Page 109
2.3.6 Participate in the System Design Review (SysDR)......Page 112
2.4.1 Preparing the Test Cycle for Subsystems and Components......Page 113
2.4.2 Assess Suppliers’ Subsystems Test Documents......Page 118
2.4.3 Perform Acceptance Test Procedure—Subsystems/Enabling Products......Page 119
2.4.4 Assess System Performance by Way of Simulation......Page 122
2.4.5 Verify Design Versus Implementation Consistency......Page 124
2.4.6 Participate in Acceptance Test Review—Subsystems/Enabling Products......Page 125
2.5.1 Develop System Integration Laboratory (SIL)......Page 126
2.5.2 Generate System Integration Test Plan (SysITP)......Page 128
2.5.3 Generate System Integration Test Description (SysITD)......Page 130
2.5.4 Validate Supplied Subsystems in Stand-Alone Configuration......Page 133
2.5.5 Perform Components, Subsystem, Enabling Products Integration Tests......Page 134
2.5.6 Generate System Integration Test Report ( SysITR)......Page 136
2.5.7 Assess Effectiveness of the System Built In Test (BIT)......Page 138
2.6 VVT ACTIVITIES DURING QUALIFICATION......Page 142
2.6.1 Generate a Qualification/Acceptance System Test Plan ( SysTP)......Page 143
2.6.2 Create Qualification/Acceptance System Test Description (SysTD)......Page 145
2.6.3 Perform Virtual System Testing by Means of Simulation......Page 147
2.6.4 Perform Qualification Testing/Acceptance Test Procedure (ATP)—System......Page 148
2.6.5 Generate Qualification/Acceptance System Test Report (SysTR)......Page 151
2.6.6 Assess System Testability, Maintainability and Availability......Page 153
2.6.7 Perform Environmental System Testing......Page 159
2.6.8 Perform System Certification and Accreditation (C&A)......Page 162
2.6.9 Conduct Test Readiness Review (TRR)......Page 166
2.6.10 Conduct Engineering Peer Review of Development Enabling Products......Page 168
2.6.11 Conduct Engineering Peer Review of Program and Project Safety......Page 170
2.7 REFERENCES......Page 171
3.1 STRUCTURE OF CHAPTER......Page 175
3.2.1 Participate in Functional Configuration Audit (FCA)......Page 176
3.2.2 Participate in Physical Configuration Audit (PCA)......Page 179
3.2.3 Plan System Production VVT Process......Page 181
3.2.4 Generate a First Article Inspection (FAI) Procedure......Page 183
3.2.6 Verify Quality of Incoming Components and Subsystems......Page 187
3.2.7 Perform First Article Inspection (FAI)......Page 188
3.2.8 Validate Pre-Production Process......Page 189
3.2.10 Perform Manufacturing Quality Control......Page 192
3.2.11 Verify the Production Operations Strategy......Page 194
3.2.12 Verify Marketing and Production Forecasting......Page 196
3.2.13 Verify Aggregate Production Planning......Page 198
3.2.14 Verify Inventory Control Operation......Page 199
3.2.15 Verify Supply Chain Management......Page 202
3.2.16 Verify Production Control Systems......Page 203
3.2.17 Verify Production Scheduling......Page 205
3.2.18 Participate in Production Readiness Review (PRR)......Page 206
3.3 VVT ACTIVITIES DURING USE/MAINTENANCE......Page 208
3.3.1 Develop VVT Plan for System Maintenance......Page 209
3.3.2 Verify the Integrated Logistics Support Plan (ILSP)......Page 213
3.3.3 Perform Ongoing System Maintenance Testing......Page 222
3.3.4 Conduct Engineering Peer Review on System Maintenance Process......Page 226
3.4 VVT ACTIVITIES DURING DISPOSAL......Page 230
3.4.1 Develop VVT Plan for System Disposal......Page 231
3.4.2 Assess the System Disposal Plan......Page 234
3.4.3 Assess System Disposal Strategies by Means of Simulation......Page 236
3.4.4 Assess On-Going System Disposal Process......Page 237
3.4.5 Conduct Engineering Peer Review to Assess System Disposal Processes......Page 241
3.5 REFERENCES......Page 243
4.2.1 Requirements Verification Matrix (RVM)......Page 245
4.2.2 System Integration Laboratory (SIL)......Page 248
4.2.3 Hierarchical VVT Optimization......Page 252
4.2.4 Defect Management and Tracking......Page 256
4.2.5 Classification Tree Method......Page 261
4.2.6 Design of Experiments (DOE)......Page 265
4.3.1 VVT Process Planning......Page 278
4.3.2 Compare Images and Documents......Page 284
4.3.3 Requirements Testability and Quality......Page 287
4.3.4 System Test Simulation......Page 294
4.3.5 Failure Mode Effect Analysis......Page 302
4.3.6 Anticipatory Failure Determination......Page 308
4.3.7 Model-Based Testing......Page 315
4.3.8 Robust Design Analysis......Page 324
4.4.1 Expert Team Reviews......Page 334
4.4.2 Formal Technical Reviews......Page 348
4.4.3 Group Evaluation and Decision......Page 353
4.5 REFERENCES......Page 368
5.1 INTRODUCTION......Page 373
5.2.1 Component and Code Coverage Testing......Page 378
5.2.2 Interface Testing......Page 382
5.3.1 Boundary Value Testing......Page 387
5.3.2 Decision Table Testing......Page 389
5.3.3 Finite State Machine Testing......Page 390
5.3.4 Human-System Interface Testing (HSI)......Page 395
5.4.1 Automatic Random Testing......Page 400
5.4.2 Performance Testing......Page 403
5.4.3 Recovery Testing......Page 407
5.4.4 Stress Testing......Page 408
5.5.1 Usability Testing......Page 410
5.5.2 Security Vulnerability Testing......Page 415
5.5.3 Reliability Testing......Page 424
5.5.4 Search-Based Testing......Page 432
5.5.5 Mutation Testing......Page 440
5.6.1 Environmental Stress Screening (ESS) Testing......Page 444
5.6.2 EMI/EMC Testing......Page 446
5.6.3 Destructive Testing......Page 448
5.6.4 Reactive Testing......Page 453
5.6.5 Temporal Testing......Page 458
5.7 BLACK BOX—PHASE TESTING......Page 465
5.7.1 Sanity Testing......Page 466
5.7.2 Exploratory Testing......Page 467
5.7.3 Regression Testing......Page 469
5.7.4 Component and Subsystem Testing......Page 474
5.7.5 Integration Testing......Page 477
5.7.6 Qualification Testing......Page 483
5.7.7 Acceptance Testing......Page 485
5.7.8 Certification and Accreditation Testing......Page 488
5.7.9 First Article Inspection (FAI)......Page 495
5.7.10 Production Testing......Page 499
5.7.11 Installation Testing......Page 503
5.7.12 Maintenance Testing......Page 506
5.7.13 Disposal Testing......Page 509
5.8 REFERENCES......Page 510
Part III: Modeling and Optimizing VVT Process......Page 517
6.1 PURPOSE AND BASIC CONCEPTS......Page 519
6.1.1 Historical Models for Cost of Quality......Page 520
6.1.2 Quantitative Models for Cost/Time of Quality......Page 521
6.2.1 Canonical VVT Cost Modeling......Page 522
6.2.2 Modeling VVT Strategy as a Decision Problem......Page 524
6.2.3 Modeling Appraisal Risk Cost......Page 527
6.2.4 Modeling Impact Risk Cost......Page 533
6.2.5 Modeling Total Quality Cost......Page 538
6.2.6 VVT Cost and Risk Example......Page 539
6.3.1 System/VVT Network......Page 543
6.3.2 Modeling Time of System/VVT Lifecycle......Page 546
6.3.3 Time and Risk Example......Page 550
6.4.2 General Fuzzy Logic Modeling......Page 552
6.4.3 Fuzzy Modeling of the VVT Process......Page 554
6.4.4 Fuzzy VVT Cost and Risk Estimation Example......Page 563
6.4.5 Fuzzy Logic Versus Probabilistic Modeling......Page 566
6.5 REFERENCES......Page 570
7.1 SYSTEMS’ QUALITY COSTS IN THE LITERATURE......Page 572
7.2.1 Quality Data Acquisition......Page 576
7.2.2 Quality Data Aggregation......Page 577
7.3.1 IAI/Lahav Pilot Project......Page 579
7.3.2 Obtaining Raw System and Quality Data......Page 581
7.3.3 Anchor System and Quality Data......Page 582
7.3.4 Generating the VVT Model Database......Page 583
7.4.1 Background......Page 584
7.4.2 Tool Availability......Page 585
7.5 VVT COST, TIME AND RISK OPTIMIZATION......Page 586
7.5.1 Optimizing the VVT Process......Page 587
7.5.2 Loss Function Optimization—VVT Cost......Page 591
7.5.3 Weight Optimization—VVT Cost......Page 598
7.5.4 Goal Optimization—VVT Cost......Page 602
7.5.5 Genetic Algorithm Optimization—VVT Time......Page 606
7.5.6 Genetic Multi-Domain Optimization—VVT Cost and Time......Page 618
7.6 REFERENCES......Page 622
8.1 METHODOLOGY VALIDATION USING A PILOT PROJECT......Page 626
8.1.1 VVT Cost Model Validation......Page 627
8.1.2 VVT Time Model Validation......Page 632
8.1.3 Fuzzy VVT Cost Model Validation......Page 639
8.2 OPTIMIZING THE VVT STRATEGY......Page 640
8.2.1 Analytical Optimization of Cost......Page 641
8.2.2 Cost Distribution by Phase......Page 648
8.2.3 Weight Optimization of Cost......Page 649
8.2.4 Goal Optimization of Cost......Page 653
8.2.5 MPGA Optimization for Time......Page 657
8.2.6 SSGA Optimization of Cost and Time......Page 659
8.3 IDENTIFYING AND AVOIDING SIGNIFICANT RISKS......Page 661
8.3.1 Avoiding Critical Risks......Page 662
8.3.2 Conjecture on Future Risk Scenarios......Page 664
8.4 IMPROVING SYSTEM QUALITY PROCESS......Page 666
A.1 ABOUT SYSTEST......Page 668
A.2 SYSTEST KEY PRODUCTS......Page 670
A.3 SYSTEST PILOT PROJECTS......Page 671
A.4 SYSTEST TEAM......Page 675
A.5 EC EVALUATION OF SYSTEST PROJECT......Page 677
REFERENCES......Page 678
B.1 BACKGROUND......Page 679
B.2 CREATING THE VVT-MP......Page 680
B.3.3 System Description......Page 681
B.4.1 Integrated VVT Program Schedule......Page 682
B.4.2 VVT Program Management......Page 683
B.5.1 VVT Strategy......Page 684
B.5.2 Planning VVT Activities......Page 687
B.5.3 VVT Limitations......Page 690
B.6.4 Expendables for Testing......Page 691
B.6.8 Budget Summary......Page 692
Appendix C: List of Acronyms......Page 693
Index......Page 701