IoT System Testing: An IoT Journey from Devices to Analytics and the Edge

✍ Scribed by Jon Duncan Hagar

• Publisher: Apress
• Year: 2022
• Tongue: English
• Leaves: 323
• Edition: 1
• Category: Library

✦ Synopsis

To succeed, teams must assure the quality of IoT systems. The world of technology continually moves from one hot area to another; this book considers the next explosion―of IoT―from a quality testing viewpoint. You'll first gain an introduction to the Internet of Things (IoT), V&V, and testing. Next, you'll be walked through IoT test planning and strategy over the full life cycle, including the impact of data analytics and AI. You will then delve deeper into IoT security testing and various test techniques, patterns, and more. This is followed by a detailed study of IoT software test labs, architecture, environments and AI. There are many options for testing IoT qualities based on the criticality of the software and risks involved; each option has positives, negatives, as well as cost and schedule impacts. The book will guide start-up and experienced teams into these paths and help you to improve the testing and quality assessment of IoT systems. What You Will Learn

• Understand IoT software test architecture and planning
• Master IoT security testing and test techniques
• Study IoT test lab automation and architectures
• Review the need for IoT security, data analytics, AI, Neural Networks and dependability using testing and V&V

 Who This Book Is ForReaders with basic knowledge of software development who want to learn more about IoT testing and its intricacies, as well as companies moving into the domain of IoT, and even those already deep into the IoT domain will benefit from this book.

✦ Table of Contents

1
Contents
About the Author
About the Technical Reviewer
Acknowledgments
2
Part 1: Getting Started
978-1-4842-8276-2.webp
978-1-4842-8276-2_1
Chapter 1: The Internet of Things, V&V, and Testing
IoT at a Glance
Understanding Chaos, Security, and Other Product Qualities
Why Test IoT Beyond the Device Itself
Agile vs. Traditional vs. “Who Cares About It” Development
This Book’s Audience
How to Use This Book
Reference Standards, Books, and the Internet: Context Matters
Valuable References to Start Your IoT Test Library
Summary
References
Figure Reference
978-1-4842-8276-2_2
Chapter 2: IoT Technology in Time and Space
IoT at a Glance
IoT Market Segments – Where Is IoT Now and in the Future?
A Sampling of IoT Challenges in Development and Testing
IoT Test Team Responses to the Development History of “Testing Is Dead”
Examples of How the IoT Devices Can Impact Testers
Summary
References
Figure References
978-1-4842-8276-2_3
Chapter 3: Big Picture Lessons Learned in IoT Project Test Planning
High-Level Issues at a Glance
#1: Creating and Releasing Software
#2: Understanding the IoT Lifecycle
#3: Test Tools for IoT
#4: Avoid Impacts of Rushing in Competition
#5: Advanced IoT Device Challenges
#6: Testing in the Complex World of IoT Systems and Systems of Systems
Project Test Risks: What Are They for IoT?
Getting Started with IoT Test Plans and Strategy
Introducing IoT Verification, Validation, and Testing Concepts and Standards
What Is IoT V&V in a Nutshell
Summary
References
Figure Reference
978-1-4842-8276-2_4
Chapter 4: Factors Driving IoT Testing/V&V Selection and Planning
Factor 1: Using Integrity Levels to Drive V&V/Test Planning and Strategy
Factor 2: Risk-Based DevOps and Product Maturity Testing
Risk Analysis Process for Testers
Factor 3: Organizational Ability Impacts on IoT Test Planning
“Newbie” Companies – Level 1
Companies with Experience Moving into IoT – Level 2
Companies with Hardware (Electronics) Experience – Level 3
Companies with Software Experience – Level 3
Company with Hardware-Software Experience – Level 4
Companies with Systems, Hardware, and Software Experience – Level 5
Government Organization – No Level but a Special Case
Experienced Companies with Only Consumer Product History – Level 2
Highly Experienced Company Including an IoT History – Level 5+
Factor 4: IoT Project Size and Complexity Impacts on Testing
Composite Total Scoring Factors (If You Need to Do That)
How IoT Projects Should Mix and Match Factors
Combining IoT Factors for Better Test Planning
IoT Test/V&V Cost Estimation
Summary
References
Figure References
978-1-4842-8276-2_5
Chapter 5: Beginner Keys for Starting IoT Test Planning
IoT Key 1: Have a Ubiquitous User Interface (UI)
IoT Key 2: Learning from Data Analytics
IoT Key 3: Unique and Specialized Hardware Working to Be a System with the Software
IoT Key 4: Level of V&V/Test Need for “Good Enough” IoT
IoT Key 5: Remaining Agile
IoT Key 6: Testing IoT, Systems, and Large/Complex Software (LCS)
Summary
References
978-1-4842-8276-2_6
Chapter 6: IoT Test Plan: Strategy and Architecture Introductions
Establishing the Mindset
Additional Considerations
Planning
Good Enough IoT Software and Devices
IoT Test Planning Basics
General Test Planning Outlined by Organization Classification
The Pure Startup, Single Device, and Small Team Who Are Trying to Stay Alive
Mature Groups or Growing Teams Targeting Growth of Sales
IoT Test for a System with More Devices
IoT Test Planning for Procuring Organizations: Governments, Large Corporations, and Others
Impact of AI, Data, and Analytics on IoT Test Planning
Product and Development Lifecycle Impacts on Test Planning – DevOps and Agile
Summary
References
Figure Reference
978-1-4842-8276-2_7
Chapter 7: IoT Test Planning and Strategy for Hardware and Software
Traditional Testing Overview
Planning in the Agile/DevOps IoT Test Lifecycle
What Should Development and Test Do During Agile and DevOps?
Ops Team People Skilled in Agile Test Thinking
Planning Using a Hybrid Agile Test Lifecycle
IoT Test Plan Budgeting, Estimating, and Scheduling
Estimating Schedules – A Brief Introduction to Support Cost Estimates
Estimation of Testing Size
Quality, Verification, Validation, and Testing
Test and Quality
Verification and Validation Activity
Wrap Up Test Planning: Regression Test Cases in IoT
IoT Test Planning: OTS Hardware and Software
Management Activities of the Coil
Engineering Using the Coil
Software Dev-Test-Sec-Ops Team in the Coil
IoT Test Planning: Security and Critical Quality Factors
IoT Test Planning: Introduction to Software Test Architecture (STA)
IoT Test Planning: IoT Strategy
Strategy – The Basics
IoT Test Strategy
What Is a Checklist for Strategy Selection?
What Is a Strategy for the Individual IoT Tester?
IoT System V&V Planning: Start with a Combination of Test Strategies
Hardware V&V Strategy (Chapters 8, 12, and 19)
Software V&V Strategy
Ops V&V Strategy
Hardware Test/V&V Planning (IEEE 1012)
Test Planning for Software Systems
IoT System Integration Test Planning
IoT System and System of Systems V&V/Test Planning – A Conceptual Overview
Summary
References
Figure Reference
978-1-4842-8276-2_8
Chapter 8: Planning for the IoT Tester on Environments and Testing Details
Assuring the Test Environment
Selecting the Right Test Environment
Planning for Automation in IoT Tool Environments
Data Analytics with Tools
IoT Detailed Project Test Planning (After the Master Test Plans)
Hardware Planning a Tester Should Know
Detailed Software Testing
System Test Plan Patterns a Tester Should Know
Planning Individual Tests (What All Testers Should Do Daily)
Test Planning from Operations to the End Product Life
Test Operations (Ops) Impacts on Test Planning
How Does Ops and Test Planning Change Over Time?
System Maintenance, Security, and Retirement
Ops Maintenance in Test Planning
Planning Retirement and Disposal of an IoT System
Testing Integration Factors
Planning Test IoT Operations and Maintenance (O&M) with Data Analytics and the Edge
Test Planning for Release Deployment
Factory Production: A Very Brief Introduction to Testing IoT Hardware
Test and Quality
Evolving As You Go
Minimum Activities Needed to Release an IoT Device into the Wild First Pass
Risk and Opportunity Management on an IoT Test Project
Last but Not Least, Test Documentation
Summary
References
978-1-4842-8276-2_9
Chapter 9: System Engineering Concepts in IoT Test Planning
Reviewing Basic Software Engineering (SE) Concepts
Critical Test Enabling Engineering Support Processes
SQA/QA
SCM and Testing
Software (Test) Measurements
Measurement and Status Supporting Systems Engineering and Testing
System Architecture and Design – Test Top-Level Support
System Planning Trade Study – Decision Analysis
Designing with Safety for IoT
Hardware Design Considerations
Software Design Considerations
IoT System Integration
Agile and DevOps Development Impacts
Summary
References
Figure Reference
978-1-4842-8276-2_10
Chapter 10: IoT Test Design: Frameworks, Techniques, Attacks, Patterns, and Tours
Test and Heuristics
Test Patterns
Example 1: Planning Pattern for IoT [8, 9]
Example 2: Mind Maps – Test Patterning Tool for Process Selection for IoT
Example 3: Attacks for IoT – High-Level Test Design Pattern
Example 4: Test Meta-design Pattern – Tours for IoT
Applying the Specific Tours to Project Factors
Example 5: Frameworks – The Top Level of Test Design Planning
Summary
References
978-1-4842-8276-2_11
Chapter 11: Classic IoT V&V/Test Concepts, Techniques, and Practices
Moving from Simple Testing to Providing Information on Quality
Techniques, Practices, Levels, and Types of Testing to Apply to IoT
Functional Test Design Techniques
Requirements Based
Boundary Value Analysis (BVA)
Scenario (a.k.a. Use Case and Stories) Testing
Syntax of IoT Command Structure Testing
Exploratory Testing
Structural Testing
Industrial Test Practices
Math-Based Testing for IoT System and Software
Model-Based Development and Testing/V&V – A Highly Automated and Integrated Test Practice
IoT Levels of Testing Related to Lifecycle Phases
Summary
References
978-1-4842-8276-2_12
Chapter 12: Test Approaches and Quality Assessments for IoT Agile/DevOps
Working Without a Formal IoT Test Plan (or Any Other Plans)
Test Planning for the Agile Teams
Agile Grows and Evolves (and Will Continue to Change)
Agile Hardware
Agile Security Testing
Agile Quality V&V Assessment
Summary
References
Figure References
978-1-4842-8276-2_13
Chapter 13: IoT Software Security Test Attacks and Designs
Before We Get Started – Security Data Points and Warnings
Definitions Used in This Chapter
Using Attack Surfaces in Security Test Planning
Planning Security Tests: IoT System Level Inspection and Assessment
Key Activity: Risk and Vulnerability Analysis
IoT Security Threat Risks and How to Find Them
Cybersecurity Team: Test/V&V Supports Developer-Operations (DevOps) Security
Key Activity: Cyber Kill Chain Model
Key Activity: Using a Zero-Trust Security Architecture Model to Support IoT Testing
Key Activity: Software Security Testing Cycle
Social Engineering
Penetration (Pen) Testing
Access Sensitive Information
Access Control and Password Cracking
Elevate Access Privileges
Denial of Service (DoS) Attack
Tampering Attack
Malware Attacks
Security Attacking the Off-the-Shelf (OTS) Software for Malware
Using a Table for Detection of Malware in the IoT Off-the-Shelf (OTS) Software
Security Taint Analysis
Spoofing [11]
Edge, Fog, and Cloud Attack: Security Testing Needed beyond a Single IoT Device
Historic Security Books You Should Have and Know
Security Test Tool Categories
Tools for Software Security Testing
IoT Testing Tools for Hardware
Software-Defined Security and AI – An IoT Future
Security Finding Reporting
Summary
References
978-1-4842-8276-2_14
Chapter 14: Security OWASP IoT Information Pointer and Logging Events
Intro to OWASP Top Ten Threats (As of 2022)
Summary
978-1-4842-8276-2_15
Chapter 15: Internal Security Team Penetration Test Process
Pen Test Process: A Beginning
Why Perform Pen Test?
Pen Security Attack and Risk-Based Test Planning for Systems
Full-Scale Penetration of the System
Simplified Process to Perform a Pen Test
Summary
References
978-1-4842-8276-2_16
Chapter 16: IoT Test Environment Introduction
Test Lab Lifecycle
Test Lab Refresher
IoT Lab/SIL Planning and Requirements
Preferred Test Environment with Full Integration of a Complex IoT System
The Field Test Environment, Analytics, and SIL Working Together
Deep Dive on Test Environment Cost and Schedule Introduction
Test Tool Introduction for IoT
Test Hardware Setup for SIL-Chaos Engineering Support with ZIF Connectors
Summary
References
Figure Reference
978-1-4842-8276-2_17
Chapter 17: Architectures Critical to Project Success
Cyber-Physical Systems
Environments and Architectures for IoT
Architecture Definition of Terms for This Book
Historical Reference: Architecture in Engineering and Literature
Summary
References
978-1-4842-8276-2_18
Chapter 18: Overview of IoT Software Architectures: Products and Testing Support
A Quick Look at IoT Architectures
Overview of IoT Support Architectures
Mind Maps of IoT Environments
IoT Software Test Architecture (STA) Introduction
IoT Software Test Architecture (STA) Details
Major Element: Test Plan (a.k.a. STAp)
Major Element: System/Software Under Test (SUT)
Major Element: Test Environment (STAe)
Major Element: Test Model (STAm)
Major Architecture Element: Views, Viewpoints, and Containers (STAv)
Summary
References
978-1-4842-8276-2_19
Chapter 19: IoT STA System: Software Integration Lab (SIL) Environments
Environment: Development Team Testing and Integration Support
Environment: Hardware Team Testing and Integration
Environment: Full Hardware-Software-System Integration
A Simple Integration STA SIL
A More Advanced IoT STA SIL with Rapid Integration Reconfiguration
Test Environment Supports: Simulation, Modeling, and Emulation
Full System IoT STA SIL with Simulation and Modeling
Environment: Real-World Full System Software Test
Special Environment: Security Test Sandbox
IoT Chaos Engineering “Live” in the Real World
Level of SIL Environment vs. Project Factors
Bringing STA to a Large-Scale Software Test Architecture/Environment
Summary
References
Figure Reference
978-1-4842-8276-2_20
Chapter 20: Tools for the Software System Integration Lab (SIL)
Test Lab Needs
Modeling and Requirements Management Tools
Automate, Automate, Automate
Simulation Test Tools Needed to Support IoT
Evolve into Automation Tooling for IoT Success
Evolve IoT Testing with AI Tools
Summary
References
978-1-4842-8276-2_21
Chapter 21: Environments for Independent Testing and IV&V on Large IoT Systems
Getting the Most Out of Independent Testing and IV&V
Verification
Validation
N-Version Testing Supporting Independence and IV&V
Lessons Learned in Testing, in Independence, and IV&V
Tooling Example for Testing Large IoT Systems and IV&V
When to Consider IoT Test Independence and IV&V Environments
Summary
References
978-1-4842-8276-2_22
Chapter 22: Self-Organizing Data Analytics (SODA): IoT Data Analytics, AI, and Statistics
SODA Model Examined
Defining SODA
SODA Implementation Options Using Stats, Taxonomies, and AI
Use Case Example for SODA and Edge Data Flow
Leveraging SODA for IoT Testing
AI SODA – A Near Future for IoT/Edge/Cloud Data Analytics
Summary
References
978-1-4842-8276-2_23
IoT Supporting Interface, Hardware, Platform, and Protocol Standards
Careers in IoT Testing
IoT Testing Startup Checklist
Getting Started with IoT Testing
Usability Testing
Compatibility and Integration Testing
App Localization Issue
External Connectivity Issues
Interrupt Testing
Operational Testing
Installation Test
Miscellaneous
Example of an IoT System of Systems
Index