Creating Augmented and Virtual Realities

Foreword
Preface
A Note on Terminology
Why We Wrote This Book
What We Didn’t Cover in This Book
How This Book Is Organized
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Design and Art Across Digital Realities
1. How Humans Interact with Computers
Common Term Definition
Introduction
Modalities Through the Ages: Pre-Twentieth Century
Modalities Through the Ages: Through World War II
Modalities Through the Ages: Post-World War II
Modalities Through the Ages: The Rise of Personal Computing
Modalities Through the Ages: Computer Miniaturization
Why Did We Just Go Over All of This?
Types of Common HCI Modalities
New Modalities
The Current State of Modalities for Spatial Computing Devices
Current Controllers for Immersive Computing Systems
Body Tracking Technologies
A Note on Hand Tracking and Hand Pose Recognition
Voice, Hands, and Hardware Inputs over the Next Generation

2. Designing for Our Senses, Not Our Devices
Envisioning a Future
Sensory Technology Explained
So, Who Are We Building This Future For?
The Role of Women in AI
Sensory Design
An Introduction
Five Sensory Principles
1. Intuitive Experiences Are Multisensory
2. 3D Will Be Normcore
3. Designs Become Physical Nature
4. Design for the Uncontrollable
5. Unlock the Power of Spatial Collaboration
Adobe’s AR Story
Conclusion

II. How eXtended Reality Is Changing Digital Art
3. Virtual Reality for Art
A More Natural Way of Making 3D Art
VR for Animation

4. 3D Art Optimization
Introduction
Options to Consider
Ideal Solution
Topology
Baking
Draw Calls
Using VR Tools for Creating 3D Art
Acquiring 3D Models Versus Making Them from Scratch
Summary

III. Hardware, SLAM, Tracking
5. How the Computer Vision That Makes Augmented Reality Possible Works
Who Are We?
A Brief History of AR
How and Why to Select an AR Platform
I’m a Developer, What Platform Should I Use and Why?
Performance Is Statistics
Integrating Hardware and Software
Optical Calibration
Inertial Calibration
The Future of Tracking
The Future of AR Computer Vision
Mapping
How Does Multiplayer AR Work?
What’s the Difficult Part?
How Does Relocalization Work?
What’s the State of the Art in Research (and Coming Soon to Consumer)?
Can the Relocalization Problem Really Be Solved for Consumers?
Can’t Google or Apple Just Do This?
Relocalization != Multiplayer; It’s Also Critical for…
How Is Relocalization Really Being Done Today in Apps?
Platforms
Apple’s ARKit
Some Mysteries Explained
Isn’t ARCore Just Tango-Lite?
So, Should I Build on ARCore Now?
What About Tango, Hololens, Vuforia, and Others?
Other Development Considerations
Lighting
Multiplayer AR—Why It’s Quite Difficult
How Do People Connect Through AR?
How Do AR Apps Connect to the World and Know Where They Really Are?
How Do AR Apps Understand and Connect to Things in the Real World?
The AR Cloud
What I Envision When I Think About the AR Cloud
How Big a Deal Is the AR Cloud?
So, Just What Is This AR Cloud?
Why All the “Meh” from the Press for ARKit and ARCore?
What’s Missing to Make a Great AR App?
Is Today’s Mobile Cloud up to the Job?
Is ARKit (or ARCore) Useless Without the AR Cloud?
The Dawn of the AR Cloud
The Bigger Picture—Privacy and AR Cloud Data
Glossary

IV. Creating Cross-Platform Augmented Reality and Virtual Reality
6. Virtual Reality and Augmented Reality: Cross-Platform Theory
Why Cross-Platform?
The Role of Game Engines
Understanding 3D Graphics
The Virtual Camera
Degrees of Freedom
Portability Lessons from Video Game Design
Simplifying the Controller Input
Development Step 1: Designing the Base Interface
Development Step 2: Platform Integration
Summary

7. Virtual Reality Toolkit: Open Source Framework for the Community
What Is VRTK and Why People Use It?
The History of VRTK
Welcome to the SteamVR Unity Toolkit
VRTK v4
The Future of VRTK
The Success of VRTK
Getting Started with VRTK 4

8. Three Virtual Reality and Augmented Reality Development Best Practices
Developing for Virtual Reality and Augmented Reality Is Difficult
Handling Locomotion
Locomotion in VR
Locomotion in AR
Effective Use of Audio
Audio in VR
Audio in AR
Common Interactions Paradigms
Inventory for VR
Augmented Reality Raycasts
Conclusion

V. Enhancing Data Representation: Data Visualization and Artificial Intelligence in Spatial Computing
9. Data and Machine Learning Visualization Design and Development in Spatial Computing
Introduction
Understanding Data Visualization
Principles for Data and Machine Learning Visualization in Spatial Computing
Why Data and Machine Learning Visualization Works in Spatial Computing
The Evolution of Data Visualization Design with the Emergence of XR
2D and 3D Data Represented in XR
2D Data Visualizations versus 3D Data Visualization in Spatial Computing
Interactivity in Data Visualizations in Spatial Computing
Animation
Failures in Data Visualization Design
Good Data Visualization Design Optimize 3D Spaces
“Savings in Time Feel Like Simplicity”
Data Representations, Infographics, and Interactions
What Qualifies as Data Visualization?
Defining Distinctions in Data Visualization and Big Data or Machine Learning Visualizations
How to Create Data Visualization: Data Visualization Creation Pipeline
WebXR: Building Data Visualizations for the Web
Data Visualization Challenges in XR
Data Visualization Industry Use Case Examples of Data Visualizations
3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XR
A Closer Look at Glass Brain
TVA Surg Medical Imaging VR Module
Data Visualization Is for Everyone: Open Source–Based Data Visualization in XR
Protein Data Visualization
Hands-On Tutorials: How to Create Data Visualization in Spatial Computing
How to Create Data Visualization: Resources
Conclusion
References
Resources
Data Visualization Tools
Machine Learning Visualization Tools
Data Journalism Visualizations
Glossary

10. Character AI and Behaviors
Introduction
Behaviors
Current Practice: Reactive AI
Adaptability
Complexity and Universality
Feasibility
More Intelligence in the System: Deliberative AI
Machine Learning
Reinforcement Learning
Deep Reinforcement Learning
Imitation Learning
Combining Automated Planning and Machine Learning
Applications
Conclusion
References

VI. Use Cases in Embodied Reality
11. The Virtual and Augmented Reality Health Technology Ecosystem
VR/AR Health Technology Application Design
Standard UX Isn’t Intuitive
Pick a Calm Environment
Convenience
Tutorial: Insight Parkinson’s Experiment
What Insight Does
How It Was Built
Companies
Planning and Guidance
Experiences Designed for Medical Education
Experiences Designed for Use by Patients
Proactive Health
Case Studies from Leading Academic Institutions

12. The Fan Experience: SportsXR
Introduction
Part 1: Five Key Principles of AR and VR for Sports
Nothing Is Live
Part 2: The Next Evolution of Sports Experiences
Part 3: Making the Future
Ownership
Final Thought
Conclusion

13. Virtual Reality Enterprise Training Use Cases
Introduction: The Importance of Enterprise Training
Does VR Training Work?
Use Case: Flood House Training
What Is VR Training Good for? R.I.D.E.
What Makes Good VR Training?
Spherical Video
The Benefits of Spherical Video
The Challenges of Spherical Video
Interactions with Spherical Video
Use Case: Factory Floor Training
The Role of Narrative
Use Case: Store Robbery Training
The Future of XR Training: Beyond Spherical Video
Computer Graphics
Use Case: Soft Skills Training
The Future: Photogrammetry
The Future: Light Fields
The Future: AR Training
The Future: Voice Recognition
The Future: The Ideal Training Scenario
References
Afterword

Index