Preface
Acknowledgements
Contents
In Situ Visualization for Computational Science: Background and Foundational Topics
1 The Motivation for In Situ Processing
1.1 Background: Computational Simulations and Post Hoc Processing
1.2 High-Performance Computing Trends Increasingly Require In Situ Processing
2 In Situ Systems
3 Challenges and Solutions for In Situ Processing
References
-22pt Data Reduction Techniques
Sampling for Scientific Data Analysis and Reduction
1 Introduction
2 Prior Work
3 Sampling Using Scalar Data Importance
3.1 Motivation for Generic Scalar Sampling
3.2 Methods for Scalar Field Sampling
3.3 Sample Analysis and Reconstruction
3.4 In Situ Analysis and Quality Comparison
4 Sampling Using Multivariate Association
4.1 Motivation for Multivariate Sampling
4.2 Multivariate Statistical Association-Driven Sampling
4.3 Applications of Multivariate Sampling
5 In Situ Performance
6 Discussions and Limitations
7 Future Directions and Conclusion
References
In Situ Wavelet Compression on Supercomputers for Post Hoc Exploration
1 Motivation
2 Introduction to Wavelet Compression
2.1 Overview of Using Wavelets for Compression
2.2 Basics of Wavelet Transforms
2.3 Compression Strategy Options
3 Evaluating the Effects of Wavelet Compression on Scientific Visualization
3.1 Critical Structure Identification
3.2 Pathline Integration Analysis
3.3 Shock Wave Front Rendering
4 Wavelet Compression on High-Performance Computers
4.1 Wavelets on Modern Computing Architectures
4.2 Overall I/O Impact
5 Conclusion
References
In Situ Statistical Distribution-Based Data Summarization and Visual Analysis
1 Statistical Distribution Models for Data Summarization
1.1 Non-parametric Distribution Models
1.2 Parametric Distribution Models
1.3 Advantages and Disadvantages of Different Distribution Models in the Context of In Situ Data Reduction
2 In Situ Distribution-Based Data Summarization Techniques
2.1 Local Distribution-Based In Situ Data Summarization
3 Post Hoc Visual Analyses Using Distribution-Based Data Summaries
3.1 Stochastic Feature Analysis
3.2 Feature Extraction and Tracking
3.3 Multivariate Query-Driven Analysis and Visualization
3.4 Distribution Sampling-Based Data Reconstruction
4 Demonstration of an In Situ Distribution-Guided End-to-End Application Study
4.1 Univariate Distribution Anomaly-Guided Stall Analysis
4.2 Multivariate Distribution Query-Driven Stall Exploration
4.3 Storage and Performance Evaluation
5 Discussion and Guidelines for Practitioners
5.1 Discussion
5.2 Guidelines for Practitioners
6 Additional Research Possibilities and Future Scopes
7 Conclusion
References
Exploratory Time-Dependent Flow Visualization via In Situ Extracted Lagrangian Representations
1 Introduction
2 Background and Motivation
2.1 Frames of Reference in Fluid Dynamics
2.2 Traditional Paradigm for Visualization and Analysis of Time-Dependent Vector Fields
3 Lagrangian-Based Flow Analysis
3.1 Phases of Computation
3.2 Differences Between Eulerian and Lagrangian-Based Flow Analysis
4 In Situ Extraction
4.1 In Situ Costs and Constraints
4.2 Spatial Sampling: Seed Placement
4.3 Temporal Sampling: Curve Approximation
4.4 Storage Format
5 Post Hoc Exploration
6 Efficacy of Lagrangian-Based In Situ + Post Hoc Flow Analysis
7 Discussion of State of the Art and Future Work
References
-22pt Workflows and Scheduling
Unlocking Large Scale Uncertainty Quantification with In Transit Iterative Statistics
1 Introduction
2 Uncertainty Management Methodology
2.1 Introduction
2.2 Quantiles of Simulation Outputs
2.3 Sensitivity Analysis via Sobol' Indices
3 In Transit Statistics
3.1 Moment-Based Statistics: Mean, Std, Higher Orders
3.2 Sobol' Indices
3.3 Order Statistics: Quantiles
3.4 Probability of Threshold Exceedance
4 The Melissa Framework
4.1 Melissa Architecture
5 An Illustrative Example
5.1 A Large Scale Study
5.2 Ubiquitous Statistic Interpretation
5.3 Combining Sobol' Indices
6 Conclusion
References
Decaf: Decoupled Dataflows for In Situ Workflows
1 Introduction
2 Background and Related Work
2.1 Types of In Situ Workflows
2.2 In Situ Workflow Runtimes
3 Design
3.1 Decaf Dataflow
3.2 Workflow Graph Description and Runtime Execution
3.3 Structure of Task Code
3.4 Data Model and Data Redistribution in the Dataflow
3.5 Flow Control Library
3.6 Data Contract Mechanism
4 Science Drivers
4.1 Molecular Dynamics
4.2 Cosmology
5 Conclusion
References
Parameter Adaptation In Situ: Design Impacts and Trade-Offs
1 Introduction
2 Impact of Simulation Load Balancing and Resource Allocation
2.1 MegaMol In Situ Loose Coupling
2.2 Workload Distribution
2.3 Load Balancing
3 Volumetric Depth Images
3.1 VDI Generation and Rendering
3.2 Parameters and Output Characteristics
3.3 Evaluation Results: Auto-Tuning Toward Target Characteristic
4 High-Resolution Streaming
4.1 Encoder Settings
4.2 Prediction of Compressed Tile Size and Quality
4.3 Optimization of Encoder Settings
4.4 Results
5 Visualization Load Balancing and Performance Prediction
5.1 Real-Time Performance Prediction
5.2 Offline Performance Prediction
6 Conclusion
References
Resource-Aware Optimal Scheduling of In Situ Analysis
1 Resource Requirements of In Situ Data Analysis
2 Effect of System Parameters on In Situ Analysis
3 Optimal Scheduling for Mode 1 (Same Job, Space-Division)
3.1 Problem Parameters
3.2 Problem Formulation
4 Optimal Scheduling for Mode 2 (Same Job, Time-Division)
4.1 Problem Parameters
4.2 Problem Formulation
4.3 Optimal Scheduling for Mode 3 (Different Jobs)
5 Experimental Evaluations
5.1 Results for Mode 1 (Same Job, Space-Division)
5.2 Results for Mode 2 (Same Job, Time-Division)
5.3 Results for Mode 3 (Different Jobs)
6 Conclusions
References
-22pt Tools
Leveraging Production Visualization Tools In Situ
1 Introduction
2 Libsim
2.1 Integration with Simulation
2.2 Runtime Behavior
2.3 Underlying Implementation
2.4 Use Case
3 Catalyst
3.1 Integration with Simulation
3.2 Runtime Behavior
3.3 Underlying Implementation
3.4 HPCMP CREATE-AVTM Helios Use Case
4 Conclusion
References
The Adaptable IO System (ADIOS)
1 Introduction
2 ADIOS I/O Abstraction
2.1 ADIOS Engines
2.2 Advanced Data Management Services
2.3 Discussion
2.4 Code Examples
3 Example Use Cases
3.1 Strong Coupling in a Fusion Simulation
3.2 Streaming Experimental Data
3.3 Interactive in Transit Visualization
4 Conclusion
References
Ascent: A Flyweight In Situ Library for Exascale Simulations
1 Introduction
1.1 Flyweight Design
1.2 Ascent Capabilities
1.3 Organization of This Chapter
2 Key Abstractions for Ascent
2.1 Pipelines
2.2 Scenes
2.3 Extracts
2.4 Queries
2.5 Triggers
2.6 Interactions Between Actions
3 Ascent APIs
3.1 Conduit: A Foundation for In-Memory Data Exchange
3.2 Mesh Blueprint: An In-Memory Mesh Description Interface
3.3 Control Interface
3.4 Typical Experiences Integrating Ascent
4 System Architecture
4.1 Flow: A Data-Type Agnostic Data-Flow Based Architecture
4.2 Runtime
5 Success Stories
5.1 In Situ Visualization of an Inertial Confinement Fusion (ICF) Simulation
5.2 MARBL Simulation Integration
5.3 Devil Ray Rendering
6 Additional Resources
References
The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale
1 Introduction and Overview
2 The SENSEI Generic In Situ Interface Design
2.1 SENSEI Data Model
2.2 SENSEI Interface
2.3 Data Types Supported in the SENSEI Interface
2.4 SENSEI Data Producer Coding Example
2.5 SENSEI Data Consumer Coding Example
3 SENSEI Tool Portability
3.1 Configurable Analysis Adaptor
3.2 Connecting SENSEI to Libsim, Catalyst, Ascent, or ADIOS
3.3 Coupling with User-Written Python Tools
3.4 In Situ Analysis of AMR Data
4 SENSEI In Situ Performance Analysis at Scale
4.1 Performance Impact of In Situ Processing
4.2 Cost Savings of In Situ Over Post Hoc
5 SENSEI In Situ Applications to Science Problems
5.1 In Situ Mesh Validation in Combustion Simulations
5.2 In Situ Processing and Analysis in Wind Energy Applications
6 Conclusion
References
In Situ Solutions with CinemaScience
1 Introduction
2 The Cinema Ecosystem
2.1 Simple Use Case: Cinema Image Databases
2.2 The Cinema Database
2.3 Cinema Writers
2.4 Cinema Viewers
2.5 Data Types Beyond Images
3 Analysis Algorithms
3.1 Computer Vision Framework
3.2 Statistical Methods
4 Task-Based Workflow Examples
5 Conclusion
References
-22pt New Research Results and Looking Forward
Deep Learning-Based Upscaling for In Situ Volume Visualization
1 Introduction
2 Background and Related Work
2.1 Artificial Neural Networks
2.2 Related Work in Upscaling
3 Upscaling ScenariosโImage-Based Upscaling
4 Upscaling Scenariosโ3D Spatial Upscaling
5 Upscaling ScenariosโTemporal Upscaling
6 Concluding Remarks and Future
References
Scalable CPU Ray Tracing for In Situ Visualization Using OSPRay
1 Introduction
2 OSPRay
2.1 OSPRay 2.0 Features for In Situ Visualization
2.2 OSPRay Actors and Objects
2.3 OSPRay Modules
2.4 OSPRay Devices
3 Distributed Rendering in OSPRay
3.1 The Distributed FrameBuffer and Rendering Algorithms
3.2 The Distributed API
3.3 Sharing Data with the Application
3.4 Asynchronous Rendering
3.5 Configuring the Scene Distribution Using Regions
3.6 Extending OSPRay's Distributed API with Modules
4 Scalability
5 Example Use Cases
5.1 Image Database Generation
6 Conclusion
References
Multivariate Functional Approximation of Scientific Data
1 Introduction
2 Related Work
3 In Situ Modeling of the MFA
3.1 Mathematical Background
3.2 Modeling with Fixed Size and Separable Dimensions
3.3 Local Adaptive Refinement
3.4 Modeling Scientific Data
4 Post Hoc Use of the MFA
4.1 Multidimensional Point Evaluation
4.2 High-Order Differentiation
4.3 Applications
5 Parallel Approximation and Evaluation
6 Ongoing and Future Work
References
A Simulation-Oblivious Data Transport Model for Flexible In Transit Visualization
1 Introduction
2 A Simulation-Oblivious Data Transport Model
2.1 Implementation in libIS
3 Example Use Cases
3.1 LAMMPS
3.2 Direct Numerical Simulation of Turbulent Channel Flow with Poongback
3.3 Comparison to Existing Libraries
4 Conclusion
References
Distributed Multi-tenant In Situ Analysis Using Galaxy
1 Introduction
2 Related Work
3 Galaxy Overview
3.1 Multi-tenancy
3.2 Using Galaxy In Situ
4 The Galaxy Ray Tracing Engine
4.1 Performance in Space and Time
4.2 Adapting Galaxy to Be a Sampling Engine
4.3 Ray Tracing Distributed Simulation Data
4.4 Ray Tracing with Asynchronous Work Messages
4.5 Asynchronous Rendering in Galaxy
4.6 Visualization Specification with Galaxy
4.7 Galaxy Ray Processing
4.8 Galaxy and Cinema
5 Galaxy Performance
6 Conclusion
References
Proximity Portability and in Transit, M-to-N Data Partitioning and Movement in SENSEI
1 Introduction and Overview
2 Data and Execution Model Design Considerations for M-to-N, In Transit Processing
2.1 Endpoint
2.2 Adaptor Pattern
2.3 Metadata
2.4 Partitioner
3 Proximity Portability and SENSEI's Use of Multiple Data Transport Tools
3.1 HDF5 In Transit Data Transport
3.2 libIS In Transit Data Transport
3.3 ADIOS In Transit Data Transport
4 Performance Analysis of SENSEI's M-to-N In Transit Infrastructure
4.1 Data Source: Oscillators Miniapplication
4.2 Data Source: AMReX-Based IAMR Code
5 Related Work
6 Conclusion and Future Work
References