MR Object Identification and Interaction: Fusing Object Situation Information from Heterogeneous Sources
Journal
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)
Type
journal article
Date Issued
2023-09-28
Author(s)
;
Khakim Akhunov
;
Federico Carbone
;
;
;
;
;
Kasim Sinan Yildirim
Research Team
Interaction- and Communication-based Systems (University of St. Gallen), Embedded and Networked Things Group (University of Trento) and TU Braunschweig
Abstract
The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection-including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing-are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond.
Language
English
Keywords
mixed reality
detection
identification
computer vision
HSG Classification
contribution to scientific community
Refereed
yes
Publisher
ACM
Publisher place
New York, NY, USA
Volume
7
Number
3
Start page
1
End page
26
Pages
26
Official URL
Subject(s)
Division(s)
Contact Email Address
jannisrene.strecker@unisg.ch
File(s)![Thumbnail Image]()
Name
Strecker et al_2023_MR Object Identification and Interaction.pdf
Size
12.3 MB
Format
Adobe PDF
Checksum (MD5)
1df38226ac4b87c34cf8cfa9a6aa49b4