Verification and Validation in Scientific Computing

✍ Scribed by William L. Oberkampf, Christopher J. Roy

• Publisher: Cambridge University Press
• Year: 2010
• Tongue: English
• Leaves: 790
• Edition: 1
• Category: Library

✦ Synopsis

Advances in scientific computing have made modelling and simulation an important part of the decision-making process in engineering, science, and public policy. This book provides a comprehensive and systematic development of the basic concepts, principles, and procedures for verification and validation of models and simulations. The emphasis is placed on models that are described by partial differential and integral equations and the simulations that result from their numerical solution. The methods described can be applied to a wide range of technical fields, from the physical sciences, engineering and technology and industry, through to environmental regulations and safety, product and plant safety, financial investing, and governmental regulations. This book will be genuinely welcomed by researchers, practitioners, and decision makers in a broad range of fields, who seek to improve the credibility and reliability of simulation results. It will also be appropriate either for university courses or for independent study.

✦ Table of Contents

Cover......Page 1
Half-title......Page 2
Title......Page 4
Copyright......Page 5
Dedication......Page 6
Contents......Page 8
Preface......Page 12
Acknowledgments......Page 14
1.1.1 Historical role of modeling and simulation......Page 16
1.1.2.1 Changing role of scientific computing in design, performance and safety of engineering systems......Page 18
1.1.2.2 Interaction of scientific computing and experimental investigations......Page 20
1.2.1 Computer speed and maturity of scientific computing......Page 23
1.2.2 Perspectives on credibility of scientific computing......Page 25
1.2.3.1 Quality of the analysts conducting the scientific computing......Page 27
1.2.3.3 Verification and validation activities......Page 28
1.2.3.4 Uncertainty quantification and sensitivity analyses......Page 29
1.3.1 Structure of the book......Page 30
1.3.2 Use of the book in undergraduate and graduate courses......Page 31
1.4 References......Page 32
Part I Fundamental concepts......Page 34
2.1 Development of concepts and terminology......Page 36
2.1.1 Early efforts of the operations research community......Page 37
2.1.2 IEEE and related communities......Page 38
2.1.4 AIAA and ASME communities......Page 40
2.1.4.1 AIAA Guide......Page 41
2.1.4.2 ASME Guide......Page 44
2.1.5 Hydrology community......Page 46
2.2.1 Code verification......Page 47
2.2.2 Solution verification......Page 49
2.2.3 Model validation......Page 50
2.2.4 Predictive capability......Page 54
2.2.5 Calibration......Page 59
2.2.6 Certification and accreditation......Page 62
2.3.1 Aleatory uncertainty......Page 66
2.3.2 Epistemic uncertainty......Page 68
2.3.2.1 Recognized uncertainty......Page 70
2.3.2.2 Blind Uncertainty......Page 71
2.4 Error in a quantity......Page 72
2.5 Integration of verification, validation, and prediction......Page 74
2.5.1 Specification of the application of interest......Page 75
2.5.2 Planning and prioritization of activities......Page 76
2.5.3 Code verification and software quality assurance activities......Page 78
2.5.4 Design and execution of validation experiments......Page 79
2.5.5 Computation of the system response quantities and solution verification......Page 81
2.5.6 Computation of validation metric results......Page 82
2.5.7 Prediction and uncertainty estimation for the application of interest......Page 84
2.5.8 Assessment of model adequacy......Page 87
2.5.9 Documentation of M&S activities......Page 88
2.6 References......Page 90
3 Modeling and computational simulation......Page 98
3.1.1 Systems and surroundings......Page 99
Example 1: Orbiting spacecraft......Page 100
Example 3: Electronic circuit......Page 101
3.1.2 Environments and scenarios......Page 102
3.2.1 Goals of scientific computing......Page 104
3.2.2 Models and simulations......Page 107
3.2.3 Importance of nondeterministic simulations......Page 111
3.2.4 Analysis of nondeterministic systems......Page 112
3.2.5.1 Aleatory uncertainty......Page 116
3.2.5.2 Epistemic uncertainty......Page 123
3.3 Risk and failure......Page 130
3.4 Phases of computational simulation......Page 131
3.4.1 Conceptual modeling phase......Page 133
3.4.2 Mathematical modeling phase......Page 134
3.4.3 Discretization and algorithm selection phase......Page 136
3.4.4 Computer programming phase......Page 138
3.4.5 Numerical solution phase......Page 139
3.4.6 Solution representation phase......Page 140
3.5 Example problem: missile flight dynamics......Page 142
3.5.1 Conceptual modeling phase......Page 143
3.5.2 Mathematical modeling phase......Page 147
3.5.4 Computer programming phase......Page 150
3.5.6 Solution representation phase......Page 151
3.6 References......Page 152
Part II Code verification......Page 160
4 Software engineering......Page 161
4.1.1 Software process models......Page 162
4.1.3 Programming languages......Page 163
4.1.4 Agile programming......Page 165
4.2 Version control......Page 166
4.3.1 Definitions......Page 168
4.3.2.2 Compiling the code......Page 169
Unit testing......Page 170
Component testing......Page 171
4.3.3.3 Software validation testing......Page 172
4.3.5 Code coverage......Page 173
4.4 Software quality and reliability......Page 174
4.4.1.2 Complexity analysis......Page 175
4.5 Case study in reliability: the T experiments......Page 176
4.6.1 Software requirements......Page 177
4.6.1.2 Requirements engineering process......Page 178
4.6.2.1 Project management......Page 179
4.6.2.3 Configuration management......Page 180
4.6.2.5 Process improvement......Page 181
4.7 References......Page 182
5 Code verification......Page 185
5.1 Code verification criteria......Page 186
5.1.1.2 Conservation tests......Page 187
5.1.2 Code-to-code comparisons......Page 188
5.1.5 Order-of-accuracy tests......Page 189
5.2.1 Truncation error......Page 190
5.2.1.1 Example: truncation error analysis......Page 191
5.2.1.2 Generalized truncation error expression (GTEE)......Page 192
5.2.3 Consistency......Page 193
5.2.5 Convergence......Page 194
5.3.1 Formal order of accuracy......Page 195
5.3.2 Observed order of accuracy......Page 197
5.4 Systematic mesh refinement......Page 200
5.4.2 Consistent mesh refinement......Page 201
5.4.3 Mesh transformations......Page 204
5.4.4 Mesh topology issues......Page 205
5.5 Order verification procedures......Page 207
2 Choose numerical algorithm......Page 208
3 Establish formal order of accuracy......Page 209
6 Compute observed order of accuracy......Page 210
5.5.2 Temporal discretization......Page 211
5.5.3.1 Separate order analysis......Page 212
5.5.3.2 Combined order analysis......Page 213
5.5.5 Limitations of order verification......Page 216
5.5.6.1 Residual method......Page 217
5.5.6.3 Downscaling method......Page 218
5.6 Responsibility for code verification......Page 219
5.7 References......Page 220
6 Exact solutions......Page 223
6.1 Introduction to differential equations......Page 224
6.2 Traditional exact solutions......Page 225
6.2.1.1 Separation of variables......Page 226
6.2.1.2 Transformations......Page 227
6.2.1.3 Method of characteristics......Page 228
6.2.3 Example with order verification: steady Burgers’ equation......Page 229
6.2.4 Example with order verification: linear elasticity......Page 232
6.3 Method of manufactured solutions (MMS)......Page 234
6.3.1 Procedure......Page 235
6.3.1.1 Manufactured solution guidelines for code verification......Page 237
6.3.1.2 Boundary and initial conditions......Page 238
6.3.2 Benefits of MMS for code verification......Page 240
6.3.3 Limitations of MMS for code verification......Page 241
6.3.4.1 2-D steady heat conduction......Page 243
6.3.4.2 2D Steady Euler equations......Page 245
6.4 Physically realistic manufactured solutions......Page 249
6.4.2 Method of nearby problems (MNP)......Page 250
6.4.2.1 Procedure......Page 251
6.4.2.2 Example exact solution: 2-D steady Navier–Stokes equations......Page 253
6.5 Approximate solution methods......Page 254
6.5.2 Reduction to ordinary differential equations......Page 255
6.5.4 Example series solution: 2-D steady heat conduction......Page 256
6.5.5 Example benchmark convergence test: 2-D hypersonic flow......Page 258
6.6 References......Page 259
Part III Solution verification......Page 264
7.1 Elements of solution verification......Page 265
7.2.1 Floating point representation......Page 267
7.2.2 Specifying precision in a code......Page 269
7.2.2.1 C/C++ programming languages......Page 270
7.2.2.2 Fortran 95/2003 programming languages......Page 271
7.2.2.3 MATLAB® Programming Language......Page 272
7.3 Statistical sampling error......Page 273
7.3.1 Estimation of statistical sampling error......Page 274
7.4 Iterative error......Page 275
7.4.1 Iterative methods......Page 276
7.4.1.1 Equations with a single unknown......Page 277
7.4.1.2 Systems of equations......Page 278
Direct solution methods......Page 280
Stationary iterative methods......Page 281
Krylov subspace methods......Page 283
Examples of iterative methods in scientific computing......Page 284
7.4.2 Iterative convergence......Page 288
Monotone convergence......Page 289
General convergence......Page 290
7.4.2.2 Iterative convergence criteria......Page 291
Iterative residuals......Page 292
7.4.3 Iterative error estimation......Page 293
Monotone iterative convergence......Page 294
7.4.4 Relation between iterative residuals and iterative error......Page 296
7.4.5 Practical approach for estimating iterative error......Page 297
7.5 Numerical error versus numerical uncertainty......Page 298
7.6 References......Page 299
8 Discretization error......Page 301
8.1.1 Discretization of the mathematical model......Page 303
8.1.1.1 The finite difference method......Page 304
8.1.1.2 The finite volume method......Page 305
8.1.1.3 The finite element method......Page 307
8.1.2.1 Structured meshes......Page 309
8.1.2.3 Cartesian meshes......Page 310
8.1.2.4 Mesh-free methods......Page 311
8.2 Approaches for estimating discretization error......Page 312
8.2.1.1 Mesh refinement methods......Page 314
8.2.1.3 Finite element recovery methods......Page 315
8.2.2.1 Error transport equations......Page 316
Continuous discretization error transport equation......Page 317
Approximating the truncation error......Page 318
System response quantities......Page 319
Implicit residual methods......Page 320
8.2.2.3 Adjoint methods for system response quantities......Page 321
Adjoint methods in the finite volume method......Page 323
8.3 Richardson extrapolation......Page 324
8.3.1 Standard Richardson extrapolation......Page 325
8.3.2 Generalized Richardson extrapolation......Page 326
8.3.3.2 Uniform mesh spacing......Page 327
8.3.4 Extensions......Page 328
8.3.4.3 Least squares extrapolation......Page 329
8.3.5.1 Example: Richardson extrapolation-based error estimation......Page 330
8.3.6 Advantages and disadvantages......Page 331
8.4.1 Asymptotic range......Page 332
8.4.2.1 Constant grid refinement factor......Page 333
8.4.2.3 Application to system response quantities......Page 335
8.4.2.4 Application to local quantities......Page 336
8.5 Discretization error and uncertainty......Page 337
8.6 Roache’s grid convergence index (GCI)......Page 338
8.6.1 Definition......Page 339
8.6.2 Implementation......Page 340
8.6.3.1 Least squares method......Page 341
8.6.3.2 Global averaging method......Page 342
8.6.3.3 Factor of safety method......Page 343
8.7.1 Measuring systematic mesh refinement......Page 344
8.7.2 Grid refinement factor......Page 345
8.7.3 Fractional uniform refinement......Page 346
8.7.4 Refinement vs. coarsening......Page 347
8.7.5 Unidirectional refinement......Page 348
8.8.1 Singularities and discontinuities......Page 349
8.8.4 Coarse grid error estimators......Page 352
8.9 References......Page 353
9.1 Factors affecting the discretization error......Page 358
9.1.2 1-D truncation error analysis on uniform meshes......Page 359
9.1.3 1-D truncation error analysis on nonuniform meshes......Page 360
9.1.4 Isotropic versus anisotropic mesh adaptation......Page 362
9.2.2 Discretization error......Page 364
9.2.3 Recovery methods......Page 365
9.2.4 Truncation errorresiduals......Page 366
9.2.4.2 Finite element residual-based methods......Page 367
9.2.5 Adjoint-based adaptation......Page 369
9.3 Adaptation approaches......Page 371
9.3.2.1 Local mesh refinementcoarsening (h-adaptation)......Page 373
9.3.2.2 Mesh movement (r-adaptation)......Page 374
9.4 Comparison of methods for driving mesh adaptation......Page 375
9.4.2 Exact solution......Page 376
9.4.3 Discretization approach......Page 377
9.4.4 Results......Page 378
9.5 References......Page 381
Part IV Model validation and prediction......Page 384
10 Model validation fundamentals......Page 394
10.1.1 Validation experiments vs. traditional experiments......Page 395
10.1.2 Goals and strategy of validation......Page 397
10.1.2.1 Scientific validation......Page 398
10.1.2.2 Project-oriented validation......Page 401
10.1.3 Sources of error in experiments and simulations......Page 404
10.1.4 Validation using data from traditional experiments......Page 408
10.2 Validation experiment hierarchy......Page 411
10.2.1 Characteristics of the complete system tier......Page 413
10.2.3 Characteristics of the benchmark tier......Page 414
10.2.4 Characteristics of the unit problem tier......Page 415
10.2.5 Construction of a validation hierarchy......Page 417
10.3 Example problem: hypersonic cruise missile......Page 419
10.3.2 Subsystem tier......Page 420
10.3.3 Benchmark tier......Page 421
10.3.5 Validation pyramid......Page 422
10.3.6 Final comments......Page 423
10.4.1 Conceptual difficulties......Page 424
10.4.2 Technical and practical difficulties......Page 427
10.5 References......Page 428
11.1 Guidelines for validation experiments......Page 432
11.1.1 Joint effort between analysts and experimentalists......Page 433
11.1.2 Measurement of all needed input data......Page 435
11.1.3 Synergism between computation and experiment......Page 438
11.1.4 Independence and dependence between computation and experiment......Page 439
11.1.5 Hierarchy of experimental measurements......Page 441
11.1.6 Estimation of experimental uncertainty......Page 444
11.2.1 Basic goals and description of JCEAP......Page 445
11.2.2.1 Wind tunnel conditions......Page 446
11.2.2.2 Model geometry......Page 448
11.2.2.3 Model fabrication and instrumentation......Page 450
11.2.3 Characterize boundary conditions and system data......Page 452
11.2.4 Synergism between computation and experiment......Page 455
11.2.6 Hierarchy of experimental measurements......Page 457
11.3.1 Random and systematic uncertainties......Page 460
11.3.2.1 DOE principles......Page 465
11.3.2.2 DOE analysis and results......Page 467
11.3.3.1 DOE principles......Page 471
11.3.3.2 DOE analysis and results......Page 473
11.4 Example of further computational–experimental synergism in JCEAP......Page 478
11.4.1 Assessment of computational submodels......Page 479
11.4.1.3 Thermodynamic submodel......Page 480
11.4.1.5 Outflow boundary condition assumption......Page 481
11.4.1.7 Re-evaluation of the experimental data......Page 482
11.4.2.1 Use of the flowfield calibration data......Page 483
11.4.2.2 Simulation using the nonuniform flowfield......Page 485
11.4.3 Lessons learned for validation experiments......Page 487
11.5 References......Page 488
12 Model accuracy assessment......Page 492
12.1 Elements of model accuracy assessment......Page 493
12.1.1 Methods of comparing simulations and experiments......Page 494
12.1.2 Uncertainty and error in model accuracy assessment......Page 497
12.1.3 Relationship between model accuracy assessment, calibration, and prediction......Page 499
12.2 Approaches to parameter estimation and validation metrics......Page 502
12.2.2 Hypothesis testing......Page 503
12.2.3 Bayesian updating......Page 506
12.2.4 Comparison of mean values......Page 507
12.3.1 Influence of numerical solution error......Page 509
12.3.3 Inclusion of experimental data post-processing......Page 510
12.3.4 Inclusion of experimental uncertainty estimation......Page 511
12.3.6 Exclusion of any type of adequacy implication......Page 512
12.3.7 Properties of a mathematical metric......Page 513
12.4.1 Perspectives of the present approach......Page 514
12.4.2 Development of the fundamental equations......Page 516
12.4.3 Construction of the validation metric for one condition......Page 518
12.4.4 Example problem: thermal decomposition of foam......Page 520
12.5.1 Construction of the validation metric over the range of the data......Page 523
12.5.2 Global metrics......Page 524
12.5.3 Example problem: turbulent buoyant plume......Page 525
12.6.1 Construction of the validation metric over the range of the data......Page 531
12.6.3 Example problem: thermal decomposition of foam......Page 533
12.7.2 Computation of simultaneous confidence intervals for the metric......Page 537
12.7.3 Global metrics......Page 539
12.7.4.1 Problem description......Page 540
12.7.4.3 Mathematical model......Page 541
12.7.4.4 Validation metric results......Page 543
12.7.5 Observations on the present approach......Page 546
12.8 Validation metric for comparing p-boxes......Page 547
12.8.1 Traditional methods for comparing distributions......Page 548
12.8.2.1 Discussion of p-boxes......Page 549
12.8.2.2 Validation metric for p-boxes......Page 551
12.8.3.1 u-pooling......Page 558
12.8.3.2 Statistical significance of a metric......Page 562
12.8.4 Inconsistency between experimental and simulation CDFs......Page 563
12.8.5.2 Epistemic and aleatory uncertainty in the metric......Page 567
12.9 References......Page 571
13 Predictive capability......Page 578
13.1 Step 1: identify all relevant sources of uncertainty......Page 580
13.1.1 Model inputs......Page 581
13.1.2 Model uncertainty......Page 583
13.1.3 Example problem: heat transfer through a plate......Page 584
13.2 Step 2: characterize each source of uncertainty......Page 588
13.2.1 Model input uncertainty......Page 592
13.2.2 Model uncertainty......Page 595
13.2.3.1 Model input uncertainty......Page 598
13.2.3.2 Model uncertainty......Page 603
13.3 Step 3: estimate numerical solution error......Page 607
13.3.1.2 Practical difficulties......Page 609
13.3.2.1 Temporal discretization error......Page 611
13.3.2.3 Richardson extrapolation error estimators for mesh convergence......Page 612
13.3.2.4 Practical difficulties......Page 613
13.3.3 Estimate of total numerical solution error......Page 614
13.3.4.1 Iterative and discretization error estimation......Page 616
13.3.4.2 Iterative and discretization error results......Page 620
13.4 Step 4: estimate output uncertainty......Page 622
13.4.1 Monte Carlo sampling of input uncertainties......Page 623
13.4.1.1 Monte Carlo sampling for aleatory uncertainties......Page 624
13.4.1.2 Monte Carlo sampling for combined aleatory and epistemic uncertainties......Page 629
13.4.2 Combination of input, model, and numerical uncertainty......Page 633
13.4.2.1 Combination of input and model uncertainty......Page 634
13.4.2.2 Estimation of model uncertainty using alternative plausible models......Page 638
13.4.3 Example problem: heat transfer through a solid......Page 640
13.4.3.1 Input uncertainties......Page 641
13.4.3.2 Combination of input, model, and numerical uncertainties......Page 643
13.5.1 Types of model parameter......Page 645
13.5.2 Sources of new information......Page 648
13.5.3 Approaches to parameter updating......Page 649
13.5.4 Parameter updating, validation, and predictive uncertainty......Page 651
13.5.4.1 Parameter updating......Page 652
13.5.4.2 Validation after parameter updating......Page 654
13.6 Step 6: conduct sensitivity analysis......Page 656
13.6.1 Local sensitivity analysis......Page 657
13.6.2 Global sensitivity analysis......Page 658
13.7 Example problem: thermal heating of a safety component......Page 661
13.7.2.1 Model input uncertainty......Page 663
Possible temperature dependence of material properties......Page 666
Characterization of model uncertainty......Page 668
13.7.3.1 General discussion of combining input and model uncertainty......Page 678
13.7.3.2 Combining input and model uncertainty for the thermal heating problem......Page 681
13.7.3.3 Predicted probabilities for the regulatory condition......Page 685
13.8 Bayesian approach as opposed to PBA......Page 687
13.9 References......Page 688
Part V Planning, management, and implementation issues......Page 694
14.1 Methodology for planning and prioritization......Page 696
14.1.1 Planning for a modeling and simulation project......Page 697
14.1.2 Value systems for prioritization......Page 699
14.2 Phenomena identification and ranking table (PIRT)......Page 701
14.2.1.2 Definition of the objectives of the PIRT process......Page 702
14.2.1.3 Specification of environments and scenarios......Page 703
14.2.1.4 Identification of plausible physical phenomena......Page 704
14.2.1.5 Construction of the PIRT......Page 705
14.3 Gap analysis process......Page 707
14.3.1 Construct the gap analysis table......Page 708
14.3.2 Documenting the PIRT and gap analysis processes......Page 711
14.3.3 Updating the PIRT and gap analysis......Page 712
14.4 Planning and prioritization with commercial codes......Page 713
14.5 Example problem: aircraft fire spread during crash landing......Page 714
14.6 References......Page 717
15.1 Survey of maturity assessment procedures......Page 719
15.2.1 Structure of the PCMM......Page 725
15.2.1.1 Representation and geometric fidelity......Page 726
15.2.1.2 Physics and material model fidelity......Page 727
15.2.1.3 Maturity assessment......Page 728
15.2.2 Purpose and uses of the PCMM......Page 730
15.2.3.1 Representation and geometric fidelity......Page 734
15.2.3.2 Physics and material model fidelity......Page 737
15.2.3.3 Code verification......Page 738
15.2.3.4 Solution verification......Page 739
15.2.3.5 Model validation......Page 741
15.2.3.6 Uncertainty quantification and sensitivity analysis......Page 742
15.3.1 Requirements for modeling and simulation maturity......Page 744
15.3.2 Aggregation of PCMM scores......Page 746
15.3.3 Use of the PCMM in risk-informed decision making......Page 747
15.4 References......Page 748
16.1 Needed technical developments......Page 751
16.2.1.1 Who should conduct SQA and code verification?......Page 752
16.2.1.2 Who should require SQA and code verification?......Page 754
16.2.2.1 Who should conduct solution verification?......Page 755
16.2.2.2 Who should require solution verification?......Page 756
16.2.3.1 Who should conduct validation?......Page 757
16.2.3.2 Who should require validation?......Page 758
16.2.4.1 Who should conduct nondeterministic predictions?......Page 759
16.2.4.2 Who should require nondeterministic predictions?......Page 760
16.3.1 Implementation issues......Page 761
16.3.3 Incorporation into business goals......Page 764
16.3.3.1 Intrinsic information quality......Page 765
16.3.3.2 Contextual information quality......Page 766
16.3.3.3 Representational information quality......Page 767
16.3.4 Organizational structures......Page 768
16.4 Development of databases......Page 770
16.4.1 Existing databases......Page 771
16.4.2 Recent activities......Page 772
16.4.3 Implementation issues of Databases......Page 773
16.5 Development of standards......Page 776
16.6 References......Page 778
Use static analyzers......Page 780
Use indentation for readability......Page 781
Duplicate code......Page 782
GOTO statements......Page 783
References......Page 784
Index......Page 785