Mastering ARKit: Apple’s Augmented Reality App Development Platform

✍ Scribed by Jayven Nhan

• Publisher: Apress
• Year: 2022
• Tongue: English
• Leaves: 550
• Category: Library

✦ Synopsis

Embark on a journey to build an augmented reality world. This book puts theory into practical application by building unique augmented reality apps specific to each chapter’s topic.
You’ll learn to engineer successful, well-designed, and comprehensive augmented reality applications for Apple devices. First, uncover the foundation of augmented reality on Apple platforms. Understand the building blocks and inner workings of the technologies that power augmented reality. Then, learn about building AR apps using ARKit, SceneKit, SpriteKit, RealityKit, and RealityComposer.
You’ll study augmented reality app business models, designs, and content for the ultimate user experience. Plus, you’ll cover the theories and practicalities of ARKit APIs such as surface detection, working with world maps, body motion capture, face tracking, object scanning and detecting, people occlusion, and more. Chapter by chapter, this book helps you tobecome an advanced augmented reality engineer.
By the end of the book, you’ll have the necessary mental models and tools to engineer delightful experiences in Apple’s augmented reality platforms.
What You'll Learn

• Build AR apps with SceneKit, SpriteKit, RealityKit, and RealityComposer
  
• Understand the business models that support augmented reality
  
• Incorporate surface detection, computer vision, and body motion capture in your apps
  

Who This Book Is ForiOS developers who want to expand their knowledge of ARKit. They should have a solid foundation in the Swift programming language and understand the basics of navigating Xcode.

✦ Table of Contents

Table of Contents
About the Author
About the Technical Reviewer
Chapter 1: Why Augmented Reality?
The Three A’s: Accessibility, Assistance, Automation
Real Estate
Education
Closing
Why ARKit?
Apple’s Core Values
Versus Cross-Platform Software
Riding the Technology Momentum
Apple’s Documentation
Apple’s Developer Community Documentation
The Apple Audience
Conclusion
Chapter 2: Introduction to ARKit: Under the Hood and Matrixes
What Is an Array?
What Is a Matrix?
Matrixes: Addition and Subtraction Operations
Matrix Identity
Setting Up Playground
Setting Up Scene View
Matrix Translation
Matrix Rotation
Matrix Scaling
Up Next
Graphics Frameworks for ARKit
SpriteKit
SceneKit
Metal, RealityKit, Reality Composer
Up Next
Session Life Cycles and Tracking Quality
Session Life Cycles
Not Available Tracking State
Limited Tracking State
Normal Tracking State
Tracking Quality Transitions
Relocalization Without a World Map
Relocalization with a World Map
Up Next
Chapter 3: Designing an Augmented Reality Experience
Ideas
Why Start with the User Experience?
Augmented Reality Benefits
Physical World Environment
Physical World Objects
Overlaying Virtual Objects
Idea Case Studies
Case Study 1: Virtual Shoe Store
Case Study 2: Computer Science Education
The Startup
User Environment
Scale
Table Scale
Room Scale
World Scale
Defined and Undefined Space
Augmented Reality Experience Prototyping
Physical Prototyping
Reality Composer
Conclusion
Chapter 4: Building Your First ARKit App with SceneKit
Purpose
Content
Creating a New Project
Setting Up ARKit SceneKit View
Connecting IBOutlet
Configuring ARSCNView Session
Allowing Camera Usage
Adding 3D Object to ARSCNView
Adding Tap Gesture Recognizer to ARSCNView
Removing Object from ARSCNView
Adding Multiple Objects to ARSCNView
Implementing Pan Gesture Recognizer on ARSCNView
Trying Out the Final App
Wrap Up
Chapter 5: Understanding and Implementing 3D Objects
What You’re Going to Learn
3D Objects Creation Toolsets
Online Resources for 3D Objects
SceneKit Supported Format
Getting Started with the Starter Project
Implementing a Single Node 3D Object
Adding Basic Lighting
Implementing a Multiple Nodes 3D Object
Loading USDZ into Scene View
Wrap Up
Chapter 6: Detecting Horizontal Planes and Adding 3D Objects with Raycast in SceneKit
What We Are Going to Learn
What’s a Horizontal Plane?
Let’s Begin to Build the App
Detecting Horizontal Planes
Adding Objects on Horizontal Planes
Raycasting for Physical World Surface Positioning
Refactoring for Segmented Control and Adding Additional Objects
Up Next
Chapter 7: Understanding Physics by Launching a Rocketship
Physics Body Explained
Physics Body Types
Static
Dynamic
Kinematic
Creating a Physics Body
Attaching a Static Physics Body
Attaching a Dynamic Physics Body
Applying Force
Adding SceneKit Particle System and Changing Physics Properties
Adding Swipe Gestures
Showtime
Implementing Custom Object Collision Detection Logic
Category Bitmasks
Collision Bitmasks
Contact Test Bitmasks
Implementing the Solution
Chapter 8: Light Estimation and Environment Texturing
What You’ll Implement and Learn
Getting Started
Creating a Sphere Node
Adding Light Node
Testing Light Properties
Showing/Hiding the Light Estimation Switch
Working with the Light Estimation Switch
Implementing Light Estimation
Trying Out the App Demo
Environment Texturing
Physically Based Lighting Model, Metalness, and Roughness
Chapter 9: 2D Image Recognition and Filter Modification
What You’ll Build
Getting Started
Enabling Image Recognition in ARKit
Physical Image Size
Image Properties
Setting Configuration for Image Recognition
Recognizing Images with ARImageAnchor
Testing the Demo App
Overlaying 3D Objects on Detected Images
Dynamically Create ARReferenceImage Programmatically
Chapter 10: Saving and Restoring World Mapping Data to Create a Persistence AR Experience
Getting Started
Working with ARWorldMap
Setting Up the World Map Local Document Directory
Archiving an AR World Map as Data
Saving an AR World Map Data into Your Document Directory
Loading an AR World Map Data from Your Document Directory
Setting the Scene View Configuration’s Initial World Map
Persistence AR Demo
Visualizing a World Map in 3D Space with ARPointCloud
Conclusion
Chapter 11: Advancing App with Real-Time Image Analysis, Machine Learning, and Vision
Getting Started
Creating an Image Analysis Request for Your Core ML Model
Classifying Camera’s Pixel Buffer from Video Frame
Controlling Toy Robot with Hand Gesture Analysis
Adding Toy Robot upon Plane Detection
Showtime
Chapter 12: Crafting 3D Assets
Crafting a Tower in SpriteKit
Crafting a Fireball in SceneKit
Crafting a Troop in Reality Composer
More Tools
Conclusion
Chapter 13: Creating Immersive Audio Experiences
Implementing Spatial Sound Awareness
Stereo to Mono Audio Channels Conversion Tools
Chapter 14: Working with SpriteKit and ARKit
Getting Started
Working with SKScene
What Is SKScene?
Setting Up a Custom SKScene
Randomly Placing 2D Object in 3D Space
Positioning a SpriteKit Node on Tap
Detecting Change in Time in SKScene
Detecting Touches in SKScene
Setting Up SKScene with View Controller
Setting Up the Time Label’s Timer
Creating the SpriteKit Node
Working with ARSKViewDelegate
Working with SceneGestureDelegate
Setting the Delegates
Running the Game
Conclusion
Chapter 15: Building Shared Experiences with Multipeer Connectivity
Getting Started
Implementing ARCoachingOverlayView
Setting Up MultipeerConnectivity
Handling MCSessionDelegate
Handling MCNearbyServiceBrowserDelegate
Handling MCNearbyServiceAdvertiserDelegate
Setting Up ViewController for MultipeerConnectivity
Sending Data in MCSession
Key-Value Observing (KVO)
Encoding Collaboration Data
Receiving Peer Data
Making Virtual Content with AnchorEntity
Working with ARParticipantAnchor
Discovering Peers
Joining Peers
Leaving Peers
Where to Go from Here?
Chapter 16: Face Tracking
Getting Started
Preparing Your Model Entity for ARFaceAnchor
Adding Your Anchor Entity onto ARFaceAnchor
Updating Anchor Entity from New ARFaceAnchor
Incorporating RealityKit Animoji in ARView
Setting Up RealityKit Action Handlers
Updating Your Animoji
Where to Go from Here?
Chapter 17: Reality Composer: Creating AR Content
Getting Started
Designing for Face Anchors
Making Your Animoji Using Primitive Shapes
Implementing Content Library’s 3D Models
Setting 3D Content Materials
Making Facial Features with Content Library’s 3D Models
Adding Behaviors to Objects
Renaming Behaviors
Implementing Your Custom Behaviors
Where to Go from Here?
Chapter 18: Simultaneously Integrate Face Tracking and World Tracking
Getting Started
Exporting a Reality File from a RealityKit Project
Encapsulating Custom Entity Properties
Creating the Entity’s API
Integrating the Custom Entity with the View Controller
Preparing the Custom Entity for Notification Triggers and Actions
Setting Up the Custom Entity Notification Triggers
Setting Up the Custom Entity Action Notifiers
Testing the Notification Triggers and Actions Integration
Where to Go from Here?
Chapter 19: Scanning 3D Objects
What’s a Reference Object
Environment Optimization for Object Scanning
Preparing Your Environment for Object Scanning
Making a Reference Object
Merging Two Reference Objects
Applying Transform to a Reference Object
Exporting a Referenced Object
Using Apple’s Sample Project to Export a Referenced Object
Where to Go from Here?
Chapter 20: Detecting 3D Objects
Getting Started
Adding Reference Objects to the Project
Configuring ARKit for Object Detection
Handling Objects Detection
Making the Buy Item Anchor Entity
Making the Sphere Model Entity
Making the Price Tag Text Model Entity
Making the Buy Text Model Entity
Where to Go from Here?
Chapter 21: Body Motion Capture
Understanding the Skeleton
Capturing a Person’s Body Motion in Real Time Using RealityKit
Integrating the Body Motion Capture Feature with a Rigged Model
Generating Your Rigged Model
Deciphering a Skeleton Using Lower-Level APIs to Spot Various Joints
Placing Objects Relative to Various Joints Captured on a Body
Calculating Distances Between Joints
Where to Go from Here?
Chapter 22: People Occlusion
Under the Hood of People Occlusion
Knowing People Occlusion Features
Configuring RealityKit for People Occlusion
Setting Frame Semantics for Person Segmentation with Depth
Turning Off People Occlusion for Performance
Where to Go from Here?
Chapter 23: Create System-Wide Accessible AR Content: Quick Look with USDZ
Getting Started
Conforming to QLPreviewControllerDataSource
Universal Zoom Animation with QLPreviewControllerDelegate
Determining AR Quick Look Canonical Web Page
Forcing Content Size to Origin in Preview
Where to Go from Here?
Chapter 24: Advancing AR Quick Look for Commerce on the Web
Getting Started
ARQuickLook Buttons
Deploying Website with GitHub Pages
Uploading Resources to GitHub Pages
Making a Merchant ID for Apple Pay with Web
Creating Your Certificate Signing Request (CSR) File
Generating the Apple Pay Payment Processing Certificate
Registering a Merchant Domain
Creating an Apple Pay Merchant Identity Certificate
Commerce Integration with Apple Pay
Where to Go from Here?
Chapter 25: Working with SwiftUI and ARKit
Getting Started
Understanding the Mechanism for Integrating ARKit with SwiftUI
Setting Up a Custom ARView
Making a View Model Observable Object for ARView
Creating a UIViewRepresentable to Contain ARView
Integrating the Coordinator Pattern
Displaying ARView in SwiftUI
Where to Go from Here?
Chapter 26: Record Augmented Reality Experiences with ReplayKit
Understanding the Purpose of ReplayKit
Integrating ReplayKit into the Project
Recording Augmented Reality Experiences
Handling Preview Controller Did Finish Protocol
Saving Augmented Reality Experiences
Wrapping the Recording Logic Inside IBAction
Where to Go from Here?
Index