Mixed-Reality Support for Context-Aware and Autonomous Industrial Processes
Type
applied research project
Start Date
April 1, 2021
End Date
August 1, 2023
Acronym
MRMRO
Status
completed
Keywords
Innovation
Mixed Reality
Automation
Industry
Industrial Support
Industrial Processes
Description
In this Innosuisse Project (Swiss Innovation Agency, #48342.1 IP-ICT), we improve industrial remote assistance and machine tooling processes through research-based innovation in the domains of context awareness and autonomous systems. This will support the lead implementation partner to become an international go-to provider of industrial MR solutions.
Leader contributor(s)
Partner(s)
Prof. Dr. Markus Stolze, Eastern Switzerland University of Applied Sciences
holo one AG
PWB AG
Bystronic AG
Funder
Topic(s)
Mixed Reality
Context-Awareness
Industrial Processes
Automation
Industrial Support
Innovation
Range
HSG + other universities + partners
Division(s)
13 results
Filters
Settings
Now showing 1 - 10 of 13
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, MR Object Identification and Interaction: Fusing Object Situation Information from Heterogeneous Sources(ACM, 2023-09-28); ; The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection-including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing-are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.Type:journal articleJournal:Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)Volume:7Issue:3DOI:10.1145/3610879Scopus© Citations 14 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, ShoppingCoach: Using Diminished Reality to Prevent Unhealthy Food Choices in an Offline Supermarket ScenarioNon-communicable diseases, such as obesity and diabetes, have a significant global impact on health outcomes. While governments worldwide focus on promoting healthy eating, individuals still struggle to follow dietary recommendations. Augmented Reality (AR) might be a useful tool to emphasize specific food products at the point of purchase. However, AR may also add visual clutter to an already complex supermarket environment. Instead, reducing the visual prevalence of unhealthy food products through Diminished Reality (DR) could be a viable alternative: We present Shopping-Coach, a DR prototype that identifies supermarket food products and visually diminishes them dependent on the deviation of the target product’s composition from dietary recommendations. In a study with 12 participants, we found that ShoppingCoach increased compliance with dietary recommendations from 75% to 100% and reduced decision time by 41%. These results demonstrate the promising potential of DR in promoting healthier food choices and thus enhancing public health.Type:conference contributionScopus© Citations 13 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, SOCRAR: Semantic OCR through Augmented Reality(ACM, 2022-11-11); ; ; ; To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.Type:conference paperJournal:Proceedings of the 12th International Conference on the Internet of ThingsScopus© Citations 13 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Gaze-enabled activity recognition for augmented reality feedback(2024-03-16); ; ; ; Andrew DuchowskiHead-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material.Type:journal articleJournal:Computers & GraphicsVolume:119Issue:Special Section on Eye Gaze VISAScopus© Citations 24 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Scopus© Citations 2 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, NeighboAR: Efficient Object Retrieval using Proximity-and Gaze-based Object Grouping with an AR SystemHumans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.Type:journal articleJournal:Proceedings of the ACM on Human-Computer InteractionVolume:8Issue:ETRADOI:10.1145/3655599Scopus© Citations 1 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Sharing Personalized Mixed Reality ExperiencesNowadays, people encounter personalized services predominantly on the Web using personal computers or mobile devices. The increasing capabilities and pervasiveness of Mixed Reality (MR) devices, however, prepare the ground for personalization possibilities that are increasingly interwoven with our physical reality, extending beyond these traditional devices. Such ubiquitous, personalized MR experiences bring the potential to make our lives and interactions with our environments more convenient, intuitive, and safer. However, these experiences will also be prone to amplify the known beneficial and, notably, harmful implications of personalization. For instance, the loss of shared world objects or the nourishing of "real-world filter bubbles" might have serious social and societal consequences as they could lead to increasingly isolated experienced realities. In this work, we envision different modes for the sharing of personalized MR environments to counteract these potential harms of ubiquitous personalization. We furthermore illustrate the different modes with use cases and list open questions towards this vision.Type:conference contribution - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Towards Societally Beneficial Personalized Realities: A Conceptual Foundation for Responsible Ubiquitous Personalization SystemsPersonalization of online realities is today ubiquitous to support decision making or reduce information overload. Recently, through the expanding capabilities and pervasiveness of Mixed Reality and Ubiquitous Computing technologies, we observe increasing personalization also of physical reality. This might yield more convenient, efficient and inclusive everyday interactions. However, it may readily lead to serious societal consequences such as the loss of shared worlds and the emergence of perceptual filter bubbles. To mitigate such harms while retaining the benefits of personalization, it is important to understand how ubiquitous personalization systems may operate responsibly. Responding to this need, we propose a conceptual model that overcomes the limitations of established personalization models and expands their applicable scope to physical, virtual, and hybrid environments. We validated our model in relation to existing literature and show how it provides a conceptual foundation for the analysis and study of responsible personalization systems that create individually and societally beneficial Personalized Realities.Type:conference paperScopus© Citations 9 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, GlassBoARd: A Gaze-Enabled AR Interface for Collaborative WorkRecent research on remote collaboration focuses on improving the sense of co-presence and mutual understanding among the collaborators, whereas there is limited research on using non-verbal cues such as gaze or head direction alongside their main communication channel. Our system – GlassBoARd – permits collaborators to see each other’s gaze behavior and even make eye contact while communicating verbally and in writing. GlassBoARd features a transparent shared Augmented Reality interface that is situated in-between two users, allowing face-to-face collaboration. From the perspective of each user, the remote collaborator is represented as an avatar that is located behind the GlassBoARd and whose eye movements are contingent on the remote collaborator’s instant eye movements. In three iterations, we improved the design of GlassBoARd and tested it with two use cases. Our preliminary evaluations showed that GlassBoARd facilitates an environment for conducting future user experiments to study the effect of sharing eye gaze on the communication bandwidth.Type:conference contributionScopus© Citations 11 - Some of the metrics are blocked by yourconsent settings
Item type:Publication, EToS-1: Eye Tracking on Shopfloors for User Engagement with Automation(CEUR Workshop Proceedings, 2022-04-30); ; ; Stolze, MarkusMixed Reality (MR) is becoming an integral part of many context-aware industrial applications. In maintenance and remote support operations, the individual steps of computer-supported (cooperative) work can be defined and presented to human operators through MR headsets. Tracking of eye movements can provide valuable insights into a user’s decision-making and interaction processes. Thus, our overarching goal is to better understand the visual inspection behavior of machine operators on shopfloors and to find ways to provide them with attention-aware and context-aware assistance through MR headsets that increasingly come with eye tracking (ET) as a default feature. Toward this goal, in two industrial scenarios, we used two mobile eye tracking devices and systematically compared the visual inspection behavior of novice and expert operators. In this paper we present our preliminary findings and lessons learned.Type:conference paper
