Intelligent Crowdsourced Testing

✍ Scribed by Qing Wang; Zhenyu Chen; Junjie Wang; Yang Feng

• Publisher: Springer
• Year: 2022
• Tongue: English
• Leaves: 251
• Category: Library

✦ Synopsis

In an article for Wired Magazine in 2006, Jeff Howe defined crowdsourcing as an idea for outsourcing a task that is traditionally performed by a single employee to a large group of people in the form of an open call. Since then, by modifying crowdsourcing into different forms, some of the most successful new companies on the market have used this idea to make people’s lives easier and better. On the other hand, software testing has long been recognized as a time-consuming and expensive activity. Mobile application testing is especially difficult, largely due to compatibility issues: a mobile application must work on devices with different operating systems (e.g. iOS, Android), manufacturers (e.g. Huawei, Samsung) and keypad types (e.g. virtual keypad, hard keypad). One cannot be 100% sure that, just because a tested application works well on one device, it will run smoothly on all others. Crowdsourced testing is an emerging paradigm that can improve the cost-effectiveness of software testing and accelerate the process, especially for mobile applications. It entrusts testing tasks to online crowdworkers whose diverse testing devices/contexts, experience, and skill sets can significantly contribute to more reliable, cost-effective and efficient testing results. It has already been adopted by many software organizations, including Google, Facebook, Amazon and Microsoft. This book provides an intelligent overview of crowdsourced testing research and practice. It employs machine learning, data mining, and deep learning techniques to process the data generated during the crowdsourced testing process, to facilitate the management of crowdsourced testing, and to improve the quality of crowdsourced testing.

✦ Table of Contents

Foreword
Preface
Contents
Part I Preliminary of Crowdsourced Testing
1 Introduction
1.1 Why We Need Crowdsourced Testing
1.2 Benefits of Crowdsourced Testing
1.3 Current Practice of Crowdsourced Testing
1.4 Challenges of Crowdsourced Testing and Solutions
2 Preliminaries
2.1 Crowdsourced Testing
2.1.1 General Procedure of Crowdsourced Testing
2.1.2 Important Concepts of Crowdsourced Testing
2.2 Basic Introduction to Artificial Intelligence Technology
2.2.1 Supervised Learning
2.2.1.1 Probabilistic Supervised Learning
2.2.1.2 k-Nearest-Neighbor Classification
2.2.1.3 Support-Vector Machine
2.2.1.4 Decision Tree
2.2.2 Unsupervised Learning
2.2.2.1 K-Means Clustering
2.2.2.2 Hierarchical Clustering
2.2.2.3 DBSCAN
2.2.3 Semi-Supervised Learning
2.2.3.1 Generative Models
2.2.3.2 Low-Density Separation
2.2.3.3 Laplacian Regularization
2.2.3.4 Heuristic Approaches
2.2.3.5 Self-Training
2.2.4 Conclusion
2.3 Typical Applications of Artificial Intelligence Technology
2.3.1 Natural Language Processing
2.3.2 Image Understanding
3 Book Structure
3.1 How the Book is Structured
Part II Supporting Technology for Crowdsourced Testing Workers
4 Characterization of Crowd Worker
4.1 Exploration of Crowd Worker's Characteristics
4.2 Crowd Worker's Characterization
4.2.1 Data Preprocessing
4.2.2 Activeness
4.2.3 Preference
4.2.4 Expertise
4.2.5 Device
5 Task Recommendation for Crowd Worker
5.1 Introduction
5.2 Learning-Based Personalized Task Recommendation
5.2.1 Motivation
5.2.2 Approach
5.2.2.1 Feature Extraction
5.2.2.2 Task Recommendation
5.2.2.3 Personalized Task Visualization Dashboard
5.2.3 Experiment
5.2.3.1 Experiment Design
5.2.3.2 Results and Analysis
5.2.4 Discussion
5.2.4.1 Generalization of PTRec
5.2.4.2 Objectivity vs. Fairness
Part III Supporting Technology for Crowdsourced Testing Tasks
6 Crowd Worker Recommendation for Testing Task
6.1 Introduction
6.2 Multi-Objective Crowd Worker Recommendation
6.2.1 Motivation
6.2.2 Approach
6.2.2.1 Modeling of Crowd Worker
6.2.2.2 Measurement of Four Objectives
6.2.2.3 Multi-Objective Optimization Framework
6.2.3 Experiment
6.2.3.1 Experimental Design
6.2.3.2 Results and Analysis
6.2.4 Discussion
6.2.4.1 Further Exploration of Results
6.2.4.2 Usefulness in Terms of Payout Schema
6.3 Context-Aware In-Process Crowd Worker Recommendation
6.3.1 Motivation
6.3.2 Approach
6.3.2.1 Testing Context Modeling
6.3.2.2 Learning-Based Ranking
6.3.2.3 Diversity-Based Re-Ranking
6.3.3 Experiment
6.3.3.1 Experiment Design
6.3.3.2 Results and Analysis
6.3.4 Discussion
6.3.4.1 Benefits of In-Process Recommendation
6.3.4.2 Implication of In-Process Recommendation
7 Crowdsourced Testing Task Management
7.1 Introduction
7.2 Completion-Aware Crowdsourced Testing Management
7.2.1 Motivation
7.2.2 Approach
7.2.2.1 Preprocess Data Based on Incremental Sampling Technique
7.2.2.2 Predict Total Bugs Using CRC
7.2.2.3 Predict Required Cost Using ARIMA
7.2.2.4 Apply iSENSE to Two Decision Scenarios in Crowdsourced Testing
7.2.3 Experiment
7.2.3.1 Experiment Design
7.2.3.2 Results and Analysis
7.2.4 Discussion
7.2.4.1 Best CRC Estimator for Crowdsourced Testing
7.2.4.2 Necessity for More Time-Sensitive Analytics in Crowdsourced Testing Decision Support
7.3 Improving Completion-Aware Crowdsourced Testing Management with Duplicate Tagger and Sanity Checker
7.3.1 Motivation
7.3.2 Approach
7.3.2.1 Incremental Sampling
7.3.2.2 Duplicate Tagger
7.3.2.3 CRC-Based Close Estimator
7.3.2.4 Coverage-Based Sanity Checker
7.3.3 Experiment
7.3.3.1 Experiment Design
7.3.3.2 Results and Analysis
7.3.4 Discussion
7.3.4.1 Influence of Semantic Analysis on Performance
7.3.4.2 Further Exploration of Sanity Checker
Part IV Supporting Technology for Crowdsourced Testing Results
8 Classification of Crowdsourced Testing Reports
8.1 Introduction
8.2 Domain Adaptation for Testing Report Classification
8.2.1 Motivation
8.2.2 Approach
8.2.2.1 Extracting Textual Features
8.2.2.2 Training SDA
8.2.2.3 Generating High-Level Features
8.2.2.4 Building a Classifier
8.2.3 Experiment
8.2.3.1 Experimental Desgin
8.2.3.2 Results and Analysis
8.2.4 Discussion
8.2.4.1 Why Does It Work?
8.2.4.2 Lessons Learned
8.3 Local-Based Active Classification of Testing Report
8.3.1 Approach
8.3.2 Experiment
8.3.2.1 Experimental Design
8.3.2.2 Results and Analysis
9 Duplicate Detection of Crowdsourced Testing Reports
9.1 Introduction
9.2 Combining Textual Description and Screenshot Information for Duplicate Reports Detection
9.2.1 Motivation
9.2.1.1 Motivating Example 1: Descriptions Could Be Confusing
9.2.1.2 Motivating Example 2: Screenshots Usually Lack Details
9.2.2 Approach
9.2.2.1 Extracting Screenshot Features
9.2.2.2 Extracting Textual Features
9.2.2.3 Conducting Duplicate Report Detection
9.2.3 Experiment
9.2.3.1 Experimental Design
9.2.4 Experimental Dataset
9.2.4.1 Results and Analysis
10 Prioritization of Crowdsourced Testing Reports
10.1 Introduction
10.2 Test Report Prioritization with Diversity and Risk Strategies
10.2.1 Motivation
10.2.2 Approach
10.2.2.1 Test Report Collection
10.2.2.2 Test Report Processing
10.2.2.3 Keyword Vector Modeling
10.2.2.4 Prioritization Strategy
10.2.3 Experiment
10.2.3.1 Experiment Design
10.2.3.2 Result Analysis
10.2.4 Discussion
10.2.4.1 Method Selection
10.2.4.2 Mobile Application Testing
10.2.4.3 Cost and Scalability
10.3 Multi-Objective Test Report Prioritization Using Image Understanding
10.3.1 Motivation
10.3.2 Approach
10.3.2.1 Text Processing
10.3.2.2 Screenshot Processing
10.3.2.3 Balanced Formula
10.3.2.4 Diversity-Based Prioritization
10.3.3 Experiment
10.3.3.1 Experimental Design
10.3.3.2 Results and Analysis
10.3.4 Discussion
10.3.4.1 Method Selection
10.3.4.2 Mobile Application Testing
11 Summarization of Crowdsourced Testing Reports
11.1 Introduction
11.2 Crowdsourced Test Report Aggregation and Summarization
11.2.1 Motivation
11.2.2 Approach
11.2.2.1 Aggregator
11.2.2.2 Summarizer
11.2.2.3 Implementation
11.2.3 Experiment
11.2.3.1 Experiment Design
11.2.3.2 Results and Analysis
12 Quality Assessment of Crowdsourced Testing Cases
12.1 Introduction
12.2 Assessing the Quality of Crowdsourced Test Cases Based on Fine-Grained Code Change History
12.2.1 Motivation
12.2.2 Approach
12.2.2.1 From Dynamic Code History to Time-Series
12.2.2.2 Extracting Time-Series Features
12.2.2.3 Predicting the Quality of Crowdsourced Tests
12.2.3 Experiment
12.2.3.1 Experiment Design
12.2.3.2 Results and Analysis
12.2.4 Discussion
12.2.4.1 Characteristics of Quality Test Cases
12.2.4.2 Implications to Research and Practice
Part V Conclusions and Future Perspectives
13 Conclusions
14 Perspectives
14.1 Fairness-Aware Recommendation
14.2 Guidance in Crowdsourced Testing
14.3 Data-Enabled Crowdsourced Testing
14.4 Screenshots Learning
14.5 Quality Assessment of Crowdsourced Testing
References