6DoF Dynamic Small Marker Tracking with Low-Cost, Unsynchronized, Dynamic Consumer Cameras For Augmented Reality Training
Nicholas Rewkowski, Andrei State, Henry Fuchs
Surgeons improve their skills through repetition of training tasks in order to operate on living patients, ideally receiving timely, useful, and objective performance feedback. However, objective performance measurement is currently difficult without 3D visualization, with effective surgical training apparatus being extremely expensive or limited in accessibility. This is problematic for medical students, especially in situations such as the COVID-19 pandemic in which they are needed by the community but have few ways of practicing without lab access. In this work, we propose and prototype a system for augmented reality (AR) visualization of laparoscopic training tasks using cheap and widely-compatible borescopes, which can track small objects typical of surgical training. We use forward kinematics for calibration and multi-threading to attempt synchronization in order to increase compatibility with consumer applications, resulting in an effective AR simulation with low-cost devices and consumer software, while also providing dynamic camera and marker tracking. We test the system with a typical peg transfer task on the HoloLens 1 and MagicLeap One.

A Conceptual Model for Data Collection and Analysis for AR-based Remote Collaboration Evaluation
Bernardo Marques, António J. Teixeira, Samuel Silva, João Alves, Paulo Dias, Beatriz Sousa Santos
A significant effort has been devoted to the creation of the enablingtechnology and in the proposal of novel methods to support remotecollaboration using Augmented Reality (AR), given the noveltyof the field. As the field progresses to focus on the nuances ofsupporting collaboration and with the growing number of proto-types mediated by AR, the characterization and evaluation of thecollaborative process becomes an essential, but difficult endeavor.Evaluation is particularly challenging in this multifaceted contextinvolving many aspects that may influence the way collaborationoccurs. Therefore, it is essential the existence of holistic evaluationstrategies that monitor the use and performance of the proposed solu-tions regarding the team, its members, and the technology, allowing adequate characterization and report of collaborative processes. Asa contribute, we propose a conceptual model for multi-user datacollection and analysis that monitors several collaboration aspects:individual and team performance, behaviour and level of collabora-tion, as well as contextual data in scenarios of remote collaborationusing AR-based solutions.

An Efficient Planar Bundle Adjustment Algorithm
Lipu Zhou, Daniel Koppel, Hui Ju, Frank Steinbruecker, Michael Kaess
This paper presents an efficient algorithm for the least-squares problem using the point-to-plane cost, which aims to jointly optimize depth sensor poses and plane parameters for 3D reconstruction. We call this least-squares problem Planar Bundle Adjustment (PBA), due to the similarity between this problem and the original Bundle Adjustment (BA) in visual reconstruction. As planes ubiquitously exist in the man-made environment, they are generally used as landmarks in SLAM algorithms for various depth sensors. PBA is important to reduce drift and improve the quality of the map. However, directly adopting the well-established BA framework in visual reconstruction will result in a very inefficient solution for PBA. This is because a 3D point only has one observation at a camera pose. In contrast, a depth sensor can record hundreds of points in a plane at a time, which results in a very large nonlinear least-squares problem even for a small-scale space. The main contribution of this paper is an efficient solution for the PBA problem using the point-to-plane cost. We introduce a reduced Jacobian matrix and a reduced residual vector, and prove that they can replace the original Jacobian matrix and residual vector in the generally adopted Levenberg-Marquardt (LM) algorithm. This significantly reduces the computational cost. Besides, when planes are combined with other features for 3D reconstruction, the reduced Jacobian matrix and residual vector can also replace the corresponding parts derived from planes. Our experimental results verify that our algorithm can significantly reduce the computational time compared to the solution using the traditional BA framework. In addition, our algorithm is faster, more accurate, and more robust to initialization errors compared to the start-of-the-art solution using the plane-to-plane cost [3].

An Evaluation of AR-Assisted Navigation for Search and Rescue in Underground Spaces
Doga Cagdas Demirkan, Sebnem Duzgun
In this study, we evaluated the performance of AR-assisted navigation in a real underground mine with good and limited illumination conditions as well as without the illumination considering possible search and rescue conditions. For this purpose, we utilized the Lumin SDK’s embedded spatial mapping algorithm for mapping and navigating. We used the spatial mapping algorithm to create the mesh model of the escape route and to render it with the user input into the Magic Leap One. Then we compared the spatial mapping algorithm in three different scenarios for the evacuation of an underground mine in an emergency situation. The escape route has two junctions and 30 meters (100 feet). The baseline scenarios are (i) evacuation of the mine in a fully illuminated condition, (ii) evacuation with the headlamp and (iii) without any illumination. For navigation in the third scenario, we provided an existing mesh of the underground space to the user. To assess the efficiency of the spatial mapping we tested each scenario with the rendered meshes. In the first scenario (fully illuminated route with the rendered meshes) the evacuation took 40 seconds. In the second scenario (illumination with the headlamp), the evacuation took 44 seconds. For the last scenario (no light source and hence in total darkness) the evacuation took 54 seconds. We found that AR-assisted navigation is effective for supporting search and rescue efforts in high attrition conditions of underground space.

An Image-Based Method for Measuring Strabismus in Virtual Reality
Wolfgang Andreas Mehringe, Markus Gerhard Wirth, Stefan Gradl, Luis Simon Durner, Matthias Ring, Annemarie F. Laudanski, Bjoern M Eskofier, Georg Michelson
Strabismus is a visual disorder characterized by eye misalignment. The effect of Panum's Fusional Area (PFA) compensates for small misalignments. However, prominent misalignments affect binocular vision and when present in childhood it may lead to amblyopia, a developmental disorder of the visual system. With the advent of Virtual Reality (VR) technology, novel binocular treatments to amblyopia arise in which the measurement of strabismus is crucial to correctly compensate for it. Thus, VR yields great potential due to the ability of displaying content to each eye independently. Major research in VR addresses this topic using eye-tracking while there is a paucity of research on image-based assessment methods. In this work, we propose a VR application for measuring strabismus in nine lines of sight. We conducted a study with 14 healthy participants to evaluate the system under two conditions: no strabismus and an artificial deviation induced by prism lenses. Further, we evaluated the effect of PFA on the system by measuring its extent in horizontal and vertical lines of sight. Results show significant difference between the expected deviation induced by prism lenses and the measured deviation. The existing difference within the measurements can be explained with the recorded extent of the PFA.

An Intelligent Augmented Reality Training Framework for Neonatal Endotracheal Intubation
Shang Zhao, Xiao Xiao, Qiyue Wang, Xiaoke Zhang, Wei Li, Lamia Soghier, James Hahn
Neonatal Endotracheal Intubation (ETI) is a critical resuscitation skill that requires tremendous practice of trainees before clinical exposure. However, current manikin-based training regimen is ineffective in providing satisfactory real-time procedural guidance for accurate assessment due to the lack of see-through visualization within the manikin. The training efficiency is further reduced by the limited availability of expert instructors, which inevitably results in a long learning curve for trainees. To this end, we propose an intelligent Augmented Reality (AR) training framework that provides trainees with a complete visualization of the ETI procedure for real-time guidance and assessment. Specifically, the proposed framework is capable of capturing the motions of the laryngoscope and the manikin and offer 3D see-through visualization rendered to the head-mounted display (HMD). Furthermore, an attention-based Convolutional Neural Network (CNN) model is developed to automatically assess the ETI performance from the captured motions as well as identify regions of motions that significantly contribute to the performance evaluation. Lastly, augmented user-friendly feedback is delivered with interpretable results with the ETI scoring rubric through the color-coded motion trajectory that classifies highlighted regions that need more practice. The classification accuracy of our machine learning model is 84.6%.

AR Circuit Constructor: Combining Electricity Building Blocks and Augmented Reality for Analogy-Driven Learning and Experimentation
Tobias Kreienbühl, Richard Wetzel, Naomi Burgess, Andrea Maria Schmid, Dorothee Brovelli
We present AR Circuit Constructor (ARCC), an augmented reality application to explore and inspect electric circuits for use in educational settings. Learners use tangible electricity building blocks to construct a working electric circuit. Then, they can use a tablet device for exploring the circuit in an augmented reality visualization. Learners can switch between three distinct conceptual analogies: bicycle chain, water pipes, and waterfalls. Through experimentation with different circuit configurations, learners explore different properties of electricity to ultimately improve their understanding of it. We describe the development of our application, including a qualitative user study with a group of STEM teachers. The latter allowed us to gain insights into the qualities required for such an application before it can ultimately be deployed in a classroom setting.

AR Mini-Games for Supermarkets
Urs Riedlinger, Leif Oppermann, Yücel Uzun, Constantin Brosda
We present an Augmented Reality (AR) application intended for use in supermarkets, with the primary goal to bring fun and digital engagement through AR mini-games to the customers while shopping. We believe that our approach can be extended and scaled up by integrating mini-games into existing shopping apps in the future.

ARPads: Mid-air Indirect Input for Augmented Reality
Eugenie Brasier, Olivier Chapuis, Nicolas FEREY, Jeanne VEZIEN, Caroline Appert
Interacting efficiently and comfortably with Augmented Reality (AR) headsets remains a major issue. We investigate the concept of mid-air pads as an alternative to gaze or direct hand input to control a cursor in windows anchored in the environment. ARPads allow users to control the cursor displayed in the headset screen through movements on a mid-air plane, which is not spatially aligned with the headset screen. We investigate a design space for ARPads, which takes into account the position of the pad relative to the user's body, and the orientation of the pad relative to that of the headset screen. Our study suggests that 1) indirect input can achieve the same performance as direct input while causing less fatigue than hand raycast, 2) an ARPad should be attached to the wrist or waist rather than to the thigh, and 3) the ARPad and the screen should have the same orientation.

Augmented Reality for Pack Optimization using Video and Depth Data
Mario Lorenz, Felix Pfeiffer, Philipp Klimant
Augmented Reality (AR) can be used in intra-logistics to optimize packing processes towards shipping less unused space by calculating and visualizing efficient pack schemas. This work describes the prototype of such a system using a Kinect v2. Based on the delivered RGB-D stream detection and track-ing algorithm applied particle filters, tracks boxes and articles. Further, the tracking approach can check if an article is placed correctly in the box. A qualitative assessment of the prototype in a warehouse revealed that such a system only useful for unexperienced packers.

Automatic Detection and Prediction of Cybersickness Severity using Deep Neural Networks from user's Physiological Signals
Rifatul Islam, Yonggun Lee, mehrad Jaloli, Imtiaz Muhammad Arafat, Dakai Zhu, peyman najafirad, Yufei Huang, John Quarles
Cybersickness is one of the primary challenges to the usability and acceptability of virtual reality (VR). Cybersickness can cause motion sickness-like discomforts, including disorientation, headache, nausea, and fatigue, both during and after the VR immersion. Prior research suggested a significant correlation between physiological signals and cybersickness severity, as measured by the simulator sickness questionnaire (SSQ). However, SSQ may not be suitable for automatic detection of cybersickness severity during immersion, as it is usually reported before and after the immersion. In this study, we introduced an automated approach for the detection and prediction of cybersickness severity from the user's physiological signals. We collected heart rate, breathing rate, heart rate variability, and galvanic skin response data from 31 healthy participants while immersed in a virtual roller coaster simulation. We found a significant difference in the participants' physiological signals during their cybersickness state compared to their resting baseline. We compared a support vector machine classifier and three deep neural classifiers for cybersickness severity detection and prediction in two minutes' future, given the previous two minutes of physiological signals. Our proposed simplified convolutional long short-term memory classifier achieved an accuracy of 97.44% for detecting current cybersickness severity and 87.38% for predicting future cybersickness severity from the physiological signals.

Catching the Drone – A Tangible Augmented Reality Game in Superhuman Sports
Christian Eichhorn, David A. Plecher, Sandro Weber, Gudrun Klinker, Yuta Itoh
The newly defined genre of Superhuman Sports provides several challenges, including the need to develop rich Augmented Reality (AR) game concepts with tangible interactions and augmentation. In this paper, we provide insights into a Superhuman Sports ball game, where players are able to interact in mid-air, rapidly and precisely with a smart, augmented and catchable drone ball. We describe our core concepts and a path towards a fully functional system with multiple and potentially different display solutions, ranging from smartphone-based AR to, eventually, HMD-based AR. For this AR game idea, a unique drone with a trackable cage based on LED pattern recognition has been developed. The player, as well as the drone will move swiftly during the game. To precisely estimate the 6DoF pose of the fast-moving drone in this dynamic scenario, we propose a suitable pipeline with a tracking algorithm. As foundation for the tracking, LEDs have been placed in a specific, spherical pattern on the drone cage. Furthermore, refinements based on the unique attributes of LEDs are considered.

Collaborative Augmented Reality on Smartphones via Life-long City-scale Maps
Lukas Platinsky, Michal Szabados, Filip Hlasek, Ross Hemsley, Luca Del Pero, Andrej Pancik, Bryan Baum, Hugo Grimmett, Peter Ondruska

CollaboVR: A Reconfigurable Framework for Creative Collaboration in Virtual Reality
Zhenyi He, Ruofei Du, Ken Perlin
Writing or sketching on whiteboards is an essential part of collab-orative discussions in business meetings, reading groups, designsessions, and interviews. However, prior work in collaborative virtual reality (VR) systems has rarely explored the design space of multi-user layouts and interaction modes with virtual whiteboards.In this paper, we present CollaboVR, a reconfigurable framework for both co-located and geographically dispersed multi-user communication in VR. Our system unleashes users’ creativity by sharing freehand drawings, converting 2D sketches into 3D models, and generating procedural animations in real-time. To minimize the computational expense for VR clients, we leverage a cloud architecture in which the computational expensive application (Chalktalk) is hosted directly on the servers, with results being simultaneously streamed to clients. We have explored three custom layouts – integrated, mir-rored, and projective – to reduce visual clutter, increase eye contact,or adapt different use cases. To evaluate CollaboVR, we conducted a within-subject user study with 12 participants. Our findings reveal that users appreciate the custom configurations and real-time interac-tions provided by CollaboVR. We have open sourced CollaboVR at https://github.com/snowymo/CollaboVR to facilitate future research and development of natural user interfaces and real-time collaborative systems in virtual and augmented reality.

Comparing Methods for Mapping Facial Expressions to Enhance Immersive Collaboration with Signs of Emotion
Natalie Hube, Oliver Lenz, Lars Engeln, Rainer Groh, Michael Sedlmair
We present a user study comparing a pre-evaluated mapping approach with a state-of-the-art direct mapping method of facial expressions for emotion judgment in an immersive setting. At its heart, the pre-evaluated approach leverages semiotics, a theory used in linguistic. In doing so, we want to compare pre-evaluation with an approach that seeks to directly map real facial expressions onto their virtual counterparts. To evaluate both approaches, we conduct a controlled lab study with 22 participants. The results show that users are significantly more accurate in judging virtual facial expressions with pre-evaluated mapping. Additionally, participants were slightly more confident when deciding on a presented emotion. We could not find any differences regarding potential Uncanny Valley effects. However, the pre-evaluated mapping shows potential to be more convenient in a conversational scenario.

Comparison of Augmented Reality Display Techniques to Support Medical Needle Insertion
Florian Heinrich, Luisa Schwenderling, Fabian Joeres, Kai Lawonn, Christian Hansen
Augmented reality (AR) may be a useful technique to overcome issues of conventionally used navigation systems supporting medical needle insertions, like increased mental workload and complicated hand-eye coordination. Previous research primarily focused on the development of AR navigation systems designed for specific displaying devices, but differences between employed methods have not been investigated before. To this end, a user study involving a needle insertion task was conducted comparing different AR display techniques with a monitor-based approach as baseline condition for the visualization of navigation information. A video see-through stationary display, an optical see-through head-mounted display and a spatial AR projector-camera-system were investigated in this comparison. Results suggest advantages of using projected navigation information in terms of lower task completion time, lower angular deviation and affirmative subjective participant feedback. Techniques requiring the intermediate view on screens, i.e. the stationary display and the baseline condition showed less favorable results. Thus, benefits of providing AR navigation information compared to a conventionally used method could be identified. Significant objective measures results, as well as an identification of advantages and disadvantages of individual display techniques contribute to the development and design of improved needle navigation systems.

Concept for a Virtual Reality Robot Ground Simulator
Mario Lorenz, Sebastian Knopp, Philipp Klimant, Johannes Quellmalz, Dr.-Ing. Holger Schlegel
For many VR applications where natural walking is necessary, the problem of a far smaller real movement space than in VR arises. Treadmills and redirected walking are established methods for this issue. However, both are limited to even surfaces and are unable to simulate different ground properties. Here a concept for a VR robot ground simulator is presented allowing to walk on steep ground or even staircase and which can simulate different undergrounds like sand, grass, or concrete. Starting from gait parameters, the technical requirements and implementation challenges for the realization of such a VR ground simulator are given.

Designing a Multitasking Interface for Object-aware AR applications
Brandon Huynh, Jason Orlosky, Tobias Höllerer
Many researchers and industry professionals believe Augmented Reality (AR) to be the next step in personal computing. But the idea of an always-on context-aware AR device presents new and unique challenges to the way users organize multiple streams of information. What does multitasking look like, when applications may be tied to specific elements in the environment? In this exploratory study, we look at one such element: physical objects, and explore an object-centric approach to multitasking in AR. We developed 3 prototype applications that operate on a subset of objects in a simulated test environment. We performed a pilot study of our multitasking solution with a novice user, domain expert, and system expert, to develop insights into the future of AR application design.

Effects of Behavioral and Anthropomorphic Realism on Social Influence with Virtual Humans in AR
Hanseul Jun, Jeremy Bailenson
While many applications in AR will display embodied agents in scenes, there is little research examining the social influence of these AR renderings. In this experiment, we manipulated the behavioral and anthropomorphic realism of an embodied agent. Participants wore an AR headset and walked a path specified by four virtual cubes, designed to bring them close to either humans or objects rendered in AR. In addition there was a control condition with no virtual objects in the room. Participants were then asked to choose between two physical chairs to sit on---one with a virtual human or object on it, or an empty one. We examined the interpersonal distance between participants and rendered objects, physical seat choice, body rotation direction while choosing a seat, and social presence ratings. For interpersonal distance, there was an effect of anthropomorphic realism but not behavioral realism---participants left more space for human-shaped objects than for non-human objects, regardless of how real the human behaved. There were no significant differences in seat choice and rotation direction. For social presence ratings, they were higher for agents high in both behavioral and anthropomorphic realism than for other conditions. We discuss implications for the social influence theory and for the design of AR systems.

Evaluation of different Visualization Techniques for Perception- Based Alignment in Medical AR
Marc J Fischer, Christoph Leuze, Alejandro Martin-Gomez, Stephanie L Perkins, Jarrett Rosenberg, Bruce Daniel
Many Augmented Reality (AR) applications require the alignment of virtual objects to the real world; this is particularly important in medical AR scenarios where medical imaging information may be displayed directly on a patient and is used to identify the exact locations of specific anatomical structures within the body. For optical see-through AR, alignment accuracy depends both on the optical parameters of the AR display as well as the visualization parameters of the virtual model. In this paper, we explore how different static visualization techniques influence users’ ability to perform perception-based alignment in AR for breast reconstruction surgery, where surgeons must accurately identify the locations of several perforator blood vessels while planning the procedure. We conducted a pilot study in which four subjects used four different visualization techniques with varying degrees of opaqueness and brightness as well as outline contrast to align virtual replicas of the relevant anatomy to their 3D-printed counterparts. We collected quantitative scores on spatial alignment accuracy using an external tracking system and qualitative scores on user preference and perceived performance. Results indicate that the highest source of alignment error was along the depth dimension, with users consistently overestimating depth when aligning the virtual renderings. The majority of subjects preferred visualization techniques rendered with lower levels of opaqueness and brightness as well as higher outline contrast, which were also found to support more accurate alignment.

Exploring Virtual Environments by Visually Impaired Using a Mixed Reality Cane Without Visual Feedback
Lei Zhang, Kelvin W, Bin Yang, Hao Tang, Zhigang Zhu
Though virtual reality (VR) has been advanced to certain levels of maturity in recent years, the general public still cannot enjoy the benefit provided by VR, especially the population of the blind and visually impaired (BVI). Current VR accessibility applications have been developed either on expensive head-mounted displays or with extra accessories and mechanisms, which are not accessible and convenient for BVI individuals. In this paper, we present a mobile VR app that enables BVI users to access a virtual environment on an iPhone in order to build their skills of perception and recognition of the virtual environment and the virtual objects in the environment. The app uses the iPhone on a selfie stick to simulate a long cane in VR, and applies Augmented Reality (AR) techniques to track the iPhone’s real-time poses in an empty space of the real world, which is then synchronized to the long cane in the VR environment. Due to the use of mixed reality (the integration of VR & AR), we call it the Mixed Reality cane (MR Cane), which provides BVI users auditory and vibrotactile feedback whenever the virtual cane comes in contact with objects in VR. Thus, the MR Cane allows BVI individuals to interact with the virtual objects and identify approximate sizes and locations of the objects in the virtual environment. We performed preliminary user studies with blind-folded participants to investigate the effectiveness of the proposed mobile approach and the results indicate that the proposed MR Cane is effective to help BVI individuals in understanding the interaction with virtual objects and exploring 3D virtual environments. The MR Cane concept can be extended to new applications of navigation, training and entertainment for BVI individuals without more significant efforts.

Extended by Design: A Toolkit for Creation of XR Experiences
Arlindo Gomes, Lucas Silva Figueiredo, Walter F M Correia, Veronica Teichrieb, Jonysberg P. Quintino, Fabio Q. B. da Silva, Andre L M Santos, Helder de Sousa Pinho
Through the last decade, the creation of extended reality (XR) solu-tions has significantly increased due to the advent of cheaper, moreadvanced, and accessible instruments like smartphones, headsets,platforms, development kits, and engines. For instance, the num-ber of GitHub repositories for XR related projects jumped from 51in 2010 to over 15,000 in 2020. At the same time, the developercommunity approaches the creation of XR applications using inher-ited design processes and methods from past mainstream platformssuch as web, mobile, or even product design. Unfortunately, thoseplatforms do not consider the spatial aspects of these applications.In this paper, we present a revisited design process and a toolkitfocused on the challenges innate to XR, that aims to help beginnersand experienced teams in the creation of applications and interac-tions in Virtual, Augmented, and Mixed Reality. We also presenta compendium of 113 techniques and 118 guidelines and a set ofcanvases that guides users through the process, preventing them fromskipping important tasks and discoveries. At last, we present a pilotcase where we accompany a team with developers and designersrunning our process and using our toolkit for the first time, showingthe benefits of a process that strikes specific issues of XR apps.

Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression
Uttaran Bhattacharya, Nick Rewkowski, Pooja Guhan, Niall Williams, Trisha Mittal, Aniket Bera, Dinesh Manocha
We present a novel autoregression network to generate virtual agents that convey various emotions through their walking styles or gaits. Given the 3D pose sequences of a gait, our network extracts pertinent movement features and affective features from the gait. We use these features to synthesize subsequent gaits such that the virtual agents can express and transition between emotions represented as combinations of happy, sad, angry, and neutral. We incorporate multiple regularizations in the training our network to simultaneously enforce plausible movements and noticeable emotions on the virtual agents. We also integrate our approach with an AR environment using a Microsoft HoloLens and can generate emotive gaits at interactive rates to increase the social presence. We evaluate how human observers perceive both the naturalness and the emotions from the generated gaits of the virtual agents in a web-based study. Our results indicate around 89% of the users found the naturalness of the gaits satisfactory on a five-point Likert scale, and the emotions they perceived from the virtual agents are statistically similar to the intended emotions of the virtual agents. We also use our network to augment existing gait datasets with emotive gaits and will release this augmented dataset for future research in emotion prediction and emotive gait synthesis.

Ginput: Fast hi-fi prototyping of gestural interactions in virtual reality
Jose Roberto Fonseca Junior, Jader Abreu, Lucas Silva Figueiredo, José Gomes Neto, Veronica Teichrieb, Jonysberg P. Quintino, Fabio Q. B. da Silva, Andre L M Santos, Helder de Sousa Pinho
Gestural interfaces in virtual reality (VR) expand the design space for user interaction, allowing spatial metaphors with the environment and more natural and immersive experiences. Typically, machine learning approaches recognize gestures with models that rely on a large number of samples for the training phase, which is an obstacle for rapidly prototyping gestural interactions. In this paper, we propose a solution designed for hi-fi prototyping of gestures within a virtual reality environment through a high-level Domain-Specific Language (DSL), as a subset of the natural language. The proposed DSL allows non-programmer users to intuitively describe a broad domain of poses and connect them for compound gestures. Our DSL was designed to be general enough for multiple input classes, such as body tracking, hand tracking, head movement, motion controllers, and buttons. We tested our solution for wands with VR designers and developers. Results showed that the tool gives non-programmers the ability to prototype gestures with ease and refine its recognition within a few minutes.

HuTrain: a Framework for Fast Creation of Real Human Pose Datasets
Ricardo Rossiter Barioni, Willams de Lima Costa, José André Carneiro Neto, Lucas Silva Figueiredo, Veronica Teichrieb, Jonysberg P. Quintino, Fabio Q. B. da Silva, Andre L M Santos, Helder de Sousa Pinho
Image-based body tracking algorithms are useful in several scenarios, such as avatar animations and gesture interaction for VR applications. In the last few years, the best-ranked solutions presented on the state of the art of body tracking (according to the most popular datasets in the field) are intensively based on Convolutional Neural Networks (CNNs) algorithms and use large datasets for training and validation. Although these solutions achieve high precision scores while evaluated with some of these datasets, there are particular tracking challenges (for example, upside-down cases) that are not well-modeled and, therefore, not correctly tracked. Instead of lurking an all-in-one solution for all cases, we propose HuTrain, a framework for creating datasets quickly and easily. HuTrain comprises a series of steps, including automatic camera calibration, refined human pose estimation, and known dataset formats conversion. We show that, with our system, the user can generate human pose datasets, targeting specific tracking challenges for the desired application context, with no need to annotate human pose instances manually.

HydrogenAR: Interactive Data-Driven Presentation of Dispenser Reliability
Matt Whitlock, Danielle Szafir, Kenny Gruchalla
When delivering presentations to a co-located audience, we typically use slides with text and 2D graphics to complement the spoken narrative. Though presentations have largely been explored on 2D media, augmented reality (AR) allows presentation designers to add data and augmentations to existing physical infrastructure on display. This could provide a more engaging experience to the audience and support comprehension. With HydrogenAR, we present a novel application that leverages the benefits of data-driven storytelling with those of AR to explain the unique challenges of hydrogen dispenser reliability. Utilizing physical props, situated data, and virtual augmentations and animations, HydrogenAR serves as a unique presentation tool, particularly critical for stakeholders, tour groups, and VIPs. HydrogenAR is a product of multiple collaborative design iterations with a local Hydrogen Fuel research team and is evaluated through interviews with team members and a user study with end-users to evaluate the usability and quality of the interactive AR experience. Through this work, we provide design considerations for AR data-driven presentations and discuss how AR could be used for innovative content delivery beyond traditional slide-based presentations.

Industrial Augmented Reality: Concepts and User Interface Designs for Augmented Reality Maintenance Worker Support Systems
Jisu Kim, Mario Lorenz, Sebastian Knopp, Philipp Klimant
Maintenance departments of producing companies in most industrial countries are facing challenges originating from an aging workforce, increasing product variety, and the pressure to increase productivity. We present the concepts and the user interface (UI) designs for two Augmented Reality (AR) applications, which help to tackle these issues. An AR Guidance System will allow new and unexperienced staff to perform medium to highly complex maintenance tasks, which they currently incapable to. The AR Remote Service System enables techni-cians at the machine to establish a voice/video stream with an internal or external expert. The video stream can be augmented with 3D models and drawings so that problems can be solved remotely and more efficiently. A qualitative assessment with maintenance managers and technicians from three producing companies rated the AR application concept as beneficial and the UI designs as very usable.

Intention to use an interactive AR app for engineering education
Alejandro Alvarez-Marin, J Ángel Velázquez-Iturbide, Mauricio Castillo-Vergara
An interactive AR app on electrical circuits was developed. The app allows the manipulation of circuit elements, computes the voltage and amperage values using the loop method, and applies Kirchhoff's voltage law. This research aims to determine the intention of using the AR app by students. It also looks to determine if it is conditioned by how the survey is applied (online or face-to-face) or gender of students. The results show that the app is well evaluated on the intention of use by students. Regarding how the survey is applied, the attitude towards using does not present significant differences. In contrast, the students who carried out the Online survey presented a higher behavioral intention to use than those who participated in the guided laboratory. Regarding gender, women showed a higher attitude toward using and behavioral intention to use this technology, compared to men.

Landmark-based mixed-reality perceptual alignment of medical imaging data and accuracy validation in living subjects
Christoph Leuze, Supriya Sathyanarayana, Bruce Daniel, Jennifer McNab
Medical augmented reality (AR) applications where virtual renderings are aligned with the real world allow to visualize internal anatomy of the patient to a medical caregiver wearing an AR headset. Accurate alignment of virtual and real content is important for applications where the virtual rendering is used to guide the medical procedure such as a surgery. Compared to 2D AR applications, where the alignment accuracy can be directly measured on the 2D screen, 3D medical AR applications require alignment measurements using phantoms and external tracking systems. In this paper we present an approach for landmark-based alignment, validation and accuracy measurement of a 3D AR overlay of medical images on the real-world subject. This is done by performing an initial MRI scan to acquire a subject’s head scan, an AR alignment task of the virtual rendering of the head MRI data to the subject’s real-world head using virtual fiducials, and a second MRI scan to test the accuracy of the AR alignment task. We have performed these 3D medical AR alignment measurements on seven volunteers using a MagicLeap AR head-mounted display. Across all seven volunteers we measured an alignment accuracy of 4.7 ± 2.6 mm. These results suggest that such an AR application can be a valuable tool for guiding non-invasive transcranial magnetic brain stimulation treatment. The presented MRI-based accuracy validation will furthermore be an important versatile tool to establish the safety of medical AR techniques.

Locomotive and Cognitive Trade-Offs for Target-based Travel
Chengyuan Lai, Afham Aiyaz, Ryan P. McMahan
Target-based travel has become a common travel metaphor for virtual reality (VR) applications. Three of the most common target-based travel techniques include Point-and-Instant-Teleport (Teleport), Point-and-Walk-Motion (Motion), and Automatic-Walk-Motion (Automatic). We present a study that employed a dual-task methodology to investigate the user performance characteristics and cognitive loads of the three target-based travel techniques, in addition to several subjective measures. Our results indicate that the Teleport technique afforded the best user travel performances, but that the Automatic technique afforded the best cognitive load.

MiXR: A Hybrid AR Sheet Music Interface for Live Performance
Shalva A. Kohen, Carmine Elvezio, Steven Feiner
Musicians face a number of challenges when performing live, including organizing and annotating sheet music. This can be an unwieldy process, as musicians need to simultaneously read and manipulate sheet music and interact with the conductor and other musicians. Augmented Reality can provide a way to ease some of the more cumbersome aspects of live performance and practice. We present MiXR, a novel interactive system that combines an AR headset, a smartphone, and a tablet to allow performers to intuitively and efficiently manage and annotate virtual sheet music in their physical environment. We discuss our underlying motivation, the interaction techniques supported, and the system architecture.

Mobile3DRecon: Real-time Monocular 3D Reconstruction on a Mobile Phone
Xingbin Yang, Liyang Zhou, Hanqing Jiang, Zhongliang Tang, Yuanbo Wang, Feng Pan, Hujun Bao, Guofeng Zhang
We present a real-time monocular 3D reconstruction system on a mobile phone, called Mobile3DRecon. Using an embedded monocular camera, our system provides an online mesh generation capability on back end together with real-time 6DoF pose tracking on front end for users to achieve realistic AR effects and interactions on mobile phones. Unlike most existing state-of-the-art systems which produce only point cloud based 3D models online or surface mesh offline, we propose a novel online incremental mesh generation approach to achieve fast online dense surface mesh reconstruction to satisfy the demand of real-time AR applications. For each keyframe of 6DoF tracking, we perform a robust monocular depth estimation, with a multi-view semi-global matching method followed by a depth refinement post-processing. The proposed mesh generation module incrementally fuses each estimated keyframe depth map to an online dense surface mesh, which is useful for achieving realistic AR effects such as occlusions and collisions. We verify our real-time reconstruction results on two mid-range mobile platforms. The experiments with quantitative and qualitative evaluation demonstrate the effectiveness of the proposed monocular 3D reconstruction system, which can handle the occlusions and collisions between virtual objects and real scenes to achieve realistic AR effects.

Modeling Emotions for Training in Immersive Simulations (METIS): A Cross-Platform Virtual Classroom Study
Alban Delamarre, Cédric Buche, Christine Lisetti
Virtual training environments (VTEs) using immersive technology have been able to successfully provide training for technical skills. Combined with recent advances in virtual social agent technologies and in affective computing, VTEs can now also support the training of social skills. Research looking at the effects of different immersive technologies on users' experience (UX) can provide important insights about their impact on user's engagement with the technology, sense presence and co-presence. However, current studies do not address whether emotions displayed by virtual agents provide the same level of UX across different virtual reality (VR) platforms. In this study, we considered a virtual classroom simulator built for desktop computer, and adapted for an immersive VR platform (CAVE). Users interact with virtual animated disruptive students able to display facial expressions, to help them practice their classroom behavior management skills. We assessed effects of the VR platforms and of the display of facial expressions on em presence, co-presence, engagement, and believability. Results indicate that users were engaged, found the virtual students believable and felt presence and co-presence for both VR platforms. We also observed an interaction effects of facial expressions and VR platforms for co-presence (p=.018<.05).

Optical Gaze Tracking with Spatially-Sparse Single-Pixel Detectors
Richard Li, Eric Whitmire, Michael Stengel, Ben Boudaoud, Jan Kautz, David Luebke, Patel Shwetak, Kaan Akşit
Gaze tracking is an essential component of next generation displays for virtual reality and augmented reality applications. Traditional camera-based gaze trackers used in next generation displays are known to be lacking in one or multiple of the following metrics:power consumption, cost, computational complexity, estimation ac-curacy, latency, and form-factor. We propose the use of discrete photodiodes and light-emitting diodes (LEDs) as an alternative to traditional camera-based gaze tracking approaches while taking all of these metrics into consideration. We begin by developing a rendering-based simulation framework for understanding the relationship between light sources and a virtual model eyeball. Findings from this framework are used for the placement of LEDs and photodiodes. Our first prototype uses a neural network to obtain an average error rate of 2.67°at 400Hz while demanding only 16mW. By simplifying the implementation to using only LEDs, duplexed as light transceivers, and more minimal machine learning model, namely a light-weight supervised Gaussian process regression algorithm, we show that our second prototype is capable of an average error rate of 1.57° at 250 Hz using 800 mW.

Real-Time Gait Reconstruction For Virtual Reality Using a Single Sensor
Tobias Feigl, Lisa Gruner, Christopher Mutschler, Daniel Roth
Embodying users through avatars based on motion tracking and reconstruction is an ongoing challenge for VR application developers. High quality VR systems use full-body tracking or inverse kinematics to reconstruct the motion of the lower extremities and control the avatar animation. Mobile systems are limited to the motion sensing of head-mounted displays (HMDs) and typically cannot offer this. We propose an approach to reconstruct gait motions from a single head-mounted accelerometer. We train our models to map head motions to corresponding ground truth gait phases. To reconstruct leg motion, the models predict gait phases to trigger equivalent synthetic animations. We designed four models: a threshold-based, a correlation-based, a Support Vector Machine (SVM) -based and a bidirectional long-term short-term memory (BLSTM) -based model. Our experiments show that, while the BLSTM approach is the most accurate, only the correlation approach runs on a mobile VR system in real time with sufficient accuracy. Our user study with 21 test subjects examined the effects of our approach on simulator sickness and showed significantly less negative effects on disorientation.

RGB-D-E: Event Camera Calibration for Fast 6-DOF Object Tracking
Etienne Dubeau, Mathieu Garon, Benoit Debaque, Raoul de Charette, Jean-François Lalonde
Augmented reality devices require multiple sensors to perform various tasks such as localization and tracking. Currently, popular cameras are mostly frame-based (e.g. RGB and Depth) which impose a high data bandwidth and power usage. With the necessity for low power and more responsive augmented reality systems, using solely frame-based sensors imposes limits to the various algorithms that needs high frequency data from the environement. As such, event-based sensors have become increasingly popular due to their low power, bandwidth and latency, as well as their very high frequency data acquisition capabilities. In this paper, we propose, for the first time, to use an event-based camera to increase the speed of 3D object tracking in 6 degrees of freedom. This application requires handling very high object speed to convey compelling AR experiences. To this end, we propose a new system which combines a recent RGB-D sensor (Kinect Azure) with an event camera (DAVIS346). We develop a deep learning approach, which combines an existing RGB-D network along with a novel event-based network in a cascade fashion, and demonstrate that our approach significantly improves the robustness of a state-of-the-art frame-based 6-DOF object tracker using our RGB-D-E pipeline. Our code and our RGB-D-E evaluation dataset are available at https://github.com/lvsn/rgbde_tracking

Scale-aware Insertion of Virtual Objects in Monocular Videos
Songhai Zhang, Xiangli Li, Yingtian Liu, Hongbo Fu
In this paper, we propose a scale-aware method for inserting virtual objects with proper sizes into monocular videos. To tackle the scale ambiguity problem of geometry recovery from monocular videos, we estimate the global scale objects in a video with a Bayesian approach incorporating the size priors of objects, where the scene object sizes should strictly conform to the same global scale and the possibilities of global scales are maximized according to the size distribution of object categories. To do so, we propose a dataset of sizes of object categories: Metric-Tree, a hierarchical representation of sizes of more than 900 object categories with the corresponding images. To handle the incompleteness of objects recovered from videos, we propose a novel scale estimation method that extracts plausible dimensions of objects for scale optimization. Experiments have shown that our method for scale estimation performs better than the state-of-the-art methods, and has considerable validity and robustness for different video scenes. Metric-Tree has been made available at: https://metric-tree.github.io

Stepping over Obstacles with Augmented Reality based on Visual Exproprioception
Alessandro Luchetti, Edoardo Parolin, Isidro III Butaslac, Yuichiro Fujimoto, Masayuki Kanbara, Paolo Bosetti, Mariolino De Cecco, Hirokazu Kato
The purpose of this study is to analyze the different kinds of exproprioceptive visual cues on an Augmented Reality (AR) system during gait exercise, on top of understanding which cues provide the best visualizations in stepping over obstacles. The main problem for users is to understand the position of virtual objects relative to themselves. Since visual exproprioception provides information about the body position in relation to the environment, it has the possibility to yield positive effects with regards to position control and gait biomechanics in the AR system. This research was born as part of the collaboration with the staff of Takanohara Central Hospital in Japan. Twenty-seven individuals were invited to take part in the user study to test three visual interfaces. The task of the mentioned user study involves making the subjects cross and avoid virtual obstacles of different heights that come from different directions. The AR application was implemented in the experiment by using the Head-Mounted Display (HMD) Microsoft HoloLens. Data obtained from the experiment revealed that the interface projected in front of the user from a third-person point of view resulted to improvements in terms of posture, visual stimuli, and safety.

The Comfort Benefits of Gaze-Directed Steering
Chengyuan Lai, Xinyu Hu, Ann Segismundo, Ananya Phadke, Ryan P. McMahan
Spatial steering is a common virtual reality (VR) travel metaphor that affords virtual locomotion and spatial understanding. Variations of spatial steering include Gaze-, Hand-, and Torso-directed steering. We present a study that employed a dual-task methodology to investigate the user performance characteristics and cognitive loads of the three spatial steering techniques, in addition to several subjective measures. Using the two one-sided tests (TOST) procedure for dependent means, we have found that Gaze- and Hand-directed steering were statistically equivalent for travel performance and cognitive load. However, we found that Gaze-directed steering induced significantly less simulator sickness than Hand-directed steering.

Towards Eyeglass-style Holographic Near-eye Displays with Statically Expanded Eyebox
Xinxing Xia, Yunqing Guan, Andrei State, Praneeth Chakravarthula, Tat-Jen Cham, henry fuchs
Holography is perhaps the only method demonstrated so far that can achieve a wide field of view (FOV) and a compact eyeglass-style form factor for augmented reality (AR) near-eye displays (NEDs). Unfortunately, the eyebox of such NEDs is impractically small (~<1mm). In this paper, we introduce and demonstrate a design for holographic NEDs with a practical, wide eyebox of ~10mm and without any moving parts, based on holographic lenslets. In our design, a holographic optical element (HOE) based on a lenslet array was fabricated as the image combiner with expanded eyebox. A phase spatial light modulator (SLM) alters the phase of the incident laser light projected onto the HOE combiner such that the virtual image can be perceived at different focus distances, which can reduce the vergence-accommodation conflict (VAC). We have successfully implemented a benchtop prototype following the proposed design. The experimental results show effective eyebox expansion to a size of ~10mm. With further work, we hope that these design concepts can be incorporated into eyeglass-size NEDs.

User-Aided Global Registration Method using Geospatial 3D Data for Large-Scale Mobile Outdoor Augmented Reality
Simon Burkard, Frank Fuchs-Kittowski
Accurate global camera registration is a key requirement for precise AR visualizations in large-scale outdoor AR applications. Existing approaches mostly use complex image-based registration methods requiring large pre-registered databases of geo-referenced images or point clouds that are hardly applicable to large-scale areas. In this paper, we present a simple yet effective user-aided registration method that utilizes common geospatial 3D data to globally register mobile devices. For this purpose, text-based 3D geospatial data including digital 3D terrain and city models is processed into small-scale 3D meshes and displayed in a live AR view. Via two common mobile touch gestures the generated virtual models can be aligned manually to match the actual perception of the real-world environment. Experimental results show that - combined with a robust local visual-inertial tracking system - this approach enables an efficient and accurate global registration of mobile devices in various environments determining the camera attitude with less than one degree deviation while achieving a high degree of immersion through realistic occlusion behavior.

Virtual Reality in Education: A Case Study on Exploring Immersive Learning for Prisoners
Jonny Collins, Tobias Langlotz, Holger Regenbrecht
Our research presented here tries to bridge the gap between technology-oriented lab work and the praxis of introducing VR technology into difficult to deploy to contexts—in our case prisoners with high learning needs. We have developed a prototypical immersive VR application designed for delivering low-level literacy and numeracy content to illiterate adults. This development has been taken to the commercial sector and is currently under product development. The target population for the application are those currently held in a correctional facility, but who have the motivation and determination to educate themselves. In this paper we discuss the current life-cycle of this project including the development, initial tests, and an exploratory study we conducted. We conclude with a discussion of logistical issues, potential research opportunities, and current outcomes.

Walking and Teleportation in Wide-area Virtual Reality Experiences
Ehsan Sayyad, Misha Sra, Tobias Höllerer
Location-based or Out-of-Home Entertainment refers to experiences such as theme and amusement parks, laser tag and paintball arenas, roller and ice skating rinks, zoos and aquariums, or science centers and museums among many other family entertainment and cultural venues. More recently, location-based VR has emerged as a new category of out-of-home entertainment. These VR experiences can be likened to social entertainment options such as laser tag, where physical movement is an inherent part of the experience versus at-home VR experiences where physical movement often needs to be replaced by artificial locomotion techniques due to tracking space constraints. In this work, we present the first VR study to understand the impact of natural walking in a large physical space on presence and user preference. We compare it with teleportation in the same large space, since teleportation is the most commonly used locomotion technique for consumer, at-home VR. Our results show that walking was overwhelmingly preferred by the participants and teleportation leads to a significantly higher disorientation component in a standard simulator sickness questionnaire. The data also shows a trend towards higher self-reported presence for natural walking.