Preface
Organization
Insights and Requirements for Future Workflow Scheduling (Keynote)
Contents
Technical Papers
On the Feasibility of Simulation-Driven Portfolio Scheduling for Cyberinfrastructure Runtime Systems
1 Introduction
2 Related Work
3 Problem Statement, Approach, Research Questions
4 Case Study
4.1 Platform Configurations
4.2 Workflow Configurations
4.3 Algorithms
4.4 Experimental Methodology
5 Results
5.1 Diversity of One-Algorithm Approaches
5.2 Evaluation in the Ideal Case (= 1, e = 0)
5.3 Evaluation with Shorter Simulation Time Horizons (< 1)
5.4 Evaluation with Simulation Inaccuracies (e > 0)
5.5 Simulation Overhead
6 Conclusion
References
AI-Job Scheduling on Systems with Renewable Power Sources
1 Introduction
1.1 Our Methods
1.2 Our Results
1.3 Related Work
2 Scheduling Algorithms
2.1 Greedy Acceptance and BestFit Allocation
3 Workload Generation
4 Performance Ratio
5 Result Representation
6 Simulation Experiments
6.1 Single-Core Results
6.2 Multi-core Results
7 Conclusion
References
Toward Building a Digital Twin of Job Scheduling and Power Management on an HPC System
1 Introduction
2 Related Work
3 A Real-World HPC System and Its Operation
3.1 Overall System Configuration
3.2 Queue Configurations
3.3 Job Scheduling and Parameters
4 Job Scheduling Simulation of a Real-World HPC System
4.1 Extended SLURM Simulator
4.2 Node State Control
5 Simulation Accuracy of AOBA-A Job Simulator
6 Survey of Scheduling and Power Saving Parameters
6.1 Parameter Settings and Job Submission Behaviors
6.2 Parameter Survey of Job Scheduling in Off Period
6.3 Parameter Survey of Job Scheduling in Busy Period
7 Conclusions
References
Encoding for Reinforcement Learning Driven Scheduling
1 Introduction
2 Background and Related Work
2.1 Cluster Scheduling
2.2 Reinforcement Learning Driven Scheduling
2.3 Encoding Approaches
3 SEM Design
3.1 Observations
3.2 Encoding of Waiting Jobs
3.3 Encoding of System Resources
3.4 Complexity Analysis
4 Evaluation
4.1 Workload Trace
4.2 Comparison Methods
4.3 Experiment Setup
4.4 Evaluation Metrics
5 Results
5.1 Convergence Rate
5.2 Scheduling Performance
5.3 Training and Inference Speedup
5.4 Scalability Analysis
6 Conclusions
References
RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources
1 Introduction
2 Background
2.1 Process Management Interface for Exascale
2.2 PMIx Reference RunTime Environment
2.3 RADICAL-Pilot
3 Related Work
4 Experiments
4.1 Use Cases
4.2 Experiments Design
4.3 Experiments on Summit
5 Conclusions
References
RARE: Renewable Energy Aware Resource Management in Datacenters
1 Introduction
2 Background and Challenges
2.1 Reinforcement Learning (RL) and Job Scheduling
2.2 Challenges
3 Renewable Energy Datacenter Environment
3.1 State Space, Action Space, and Reward Function
3.2 Renewable Energy Forecasting
3.3 Energy Storage Devices (ESDs)
4 DRL Scheduling Agent
5 Evaluation
5.1 Experimentation Conditions
5.2 Evaluation Metrics
5.3 Workload
5.4 Results
6 Related Work
6.1 Heuristics Schedulers
6.2 RL Schedulers
7 Conclusion
References
Dynamic Management of CPU Resources Towards Energy Efficient and Profitable Datacentre Operation
1 Introduction
2 Background
2.1 System Model
2.2 VM Allocation Approach
3 Node-Level Estimations
3.1 Target CPU Utilization and VM Evictions
3.2 Load Estimation
3.3 Energy Cost Estimation
3.4 SLA Violation Cost Estimation
3.5 Total Estimated Node Cost
4 Adaptive Target CPU Utilization
5 VM Allocation and Node Configuration Policy
6 Experimental Evaluation
6.1 Experimental Setup
6.2 Hardware Parameters
6.3 Threshold and Cost Convergence
6.4 ATUC Total Cost of Operation
6.5 Comparison of ATUC with State-of-the-Art Policies
7 Related Work
8 Conclusions
References
Optimization of Execution Parameters of Moldable Ultrasound Workflows Under Incomplete Performance Data
1 Introduction
2 Automatic Optimization of Workflow Execution Parameters
2.1 k-Dispatch Workflow Management System
2.2 Workflow Optimization Within k-Dispatch
2.3 Estimator Module and Interpolation Techniques
2.4 Evaluator Module Improvement
3 Experiment Setup
3.1 Investigated Workflows
3.2 Used Datasets
3.3 Tetrisator Validation Against Real Cluster
3.4 Workflow Schedule Quality Measures
3.5 Evaluation of Interpolation Techniques
4 Experimental Results
4.1 Simulated Execution Plans Reliability
4.2 Interpolation Functions Accuracy
4.3 Impact of Interpolation on Schedule Makespan and Cost
5 Conclusions
5.1 Future Work
References
Scheduling of Elastic Message Passing Applications on HPC Systems
1 Introduction
2 Related Works
3 Application Model
4 Scheduling Algorithms
4.1 Evaluation Metrics
5 Experiment and Results
5.1 Workload
5.2 Experimental Setup
5.3 Results
5.4 Analysis
6 Conclusion and Future Works
References
Improving Accuracy of Walltime Estimates in PBS Professional Using Soft Walltimes
1 Introduction
2 Soft Walltime Adoption in CERIT-SC
2.1 Inaccurate Estimates
2.2 Soft Walltime Functionality
2.3 Soft Walltime Predictor Implementation
3 Comparison of Soft Walltime Predictors
3.1 Initial Evaluation and Applied Modifications
3.2 Analysis of Soft Walltime Accuracy
3.3 Soft Walltime Caveats
4 Comparison of System Performance
4.1 Comparison of Workload Characteristics
4.2 Comparison of Wait Time and Bounded Slowdown
5 Conclusion and Future Work
References
Re-making the Movie-Making Machine
1 Introduction
1.1 Basic Concepts
2 Background
2.1 Coda
2.2 Production Proven
2.3 Architecture
2.4 Concepts and Configuration
2.5 Scale and Complexity
2.6 Problem Statement
3 Observability
3.1 Logging
3.2 Dashboard
4 Optimisations
4.1 Thread Model
4.2 Parallelisation
4.3 Redundant Calculations
4.4 Memory Footprint
4.5 Memory Allocator
4.6 Stability
4.7 Asynchronous Events
4.8 Networking
4.9 Work Distribution
4.10 Waste Reduction
5 Simulations and Testing
5.1 Coda in a Box (CIAB)
5.2 Integration Tests
5.3 Saturation Tests
5.4 Simulation
6 Results
6.1 Simulation & Saturation
6.2 In Production
6.3 Server Resources
7 Future Work
References
Open Scheduling Problems
Using Kubernetes in Academic Environment: Problems and Approaches
1 Introduction
2 Scheduling Challenges
2.1 Endless Computing with Limited Resources
2.2 Interactive Computing
2.3 Overestimation
2.4 Problem Summary and Scheduling Objectives
3 Scheduling in Kubernetes
3.1 Pods and Jobs
3.2 Resource Requests and Limits
3.3 Priority Classes
3.4 Labels, Affinity, Anti-affinity
4 Problem Solutions Using Kubernetes Building Blocks
4.1 Separate Clusters
4.2 Scavanger Jobs
4.3 Placeholder Jobs
4.4 Unresolved Issues
5 Real Workload Trace from CERIT-SC Installation
6 Related Work
7 Conclusion
References
Author Index