Kimberly Garcia

The increasing number of objects in ubiquitous computing environments creates a need for effective object detection and identification mechanisms that permit users to intuitively initiate interactions with these objects. While multiple approaches to such object detection-including through visual object detection, fiducial markers, relative localization, or absolute spatial referencing-are available, each of these suffers from drawbacks that limit their applicability. In this paper, we propose ODIF, an architecture that permits the fusion of object situation information from such heterogeneous sources and that remains vertically and horizontally modular to allow extending and upgrading systems that are constructed accordingly. We furthermore present BLEARVIS, a prototype system that builds on the proposed architecture and integrates computer-vision (CV) based object detection with radio-frequency (RF) angle of arrival (AoA) estimation to identify BLE-tagged objects. In our system, the front camera of a Mixed Reality (MR) head-mounted display (HMD) provides a live image stream to a vision-based object detection module, while an antenna array that is mounted on the HMD collects AoA information from ambient devices. In this way, BLEARVIS is able to differentiate between visually identical objects in the same environment and can provide an MR overlay of information (data and controls) that relates to them. We include experimental evaluations of both, the CV-based object detection and the RF-based AoA estimation, and discuss the applicability of the combined RF and CV pipelines in different ubiquitous computing scenarios. This research can form a starting point to spawn the integration of diverse object detection, identification, and interaction approaches that function across the electromagnetic spectrum, and beyond. CCS Concepts: • Human-centered computing → Mixed / augmented reality; Ubiquitous and mobile computing systems and tools; • Hardware → Radio frequency and wireless interconnect.

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they allow novel interaction methods and provide insights into user attention, intentions, and activities. However, only few studies have used gaze-enabled AR displays for human activity recognition (HAR). In an experimental study, we collected gaze data from 10 users on a HoloLens 2 (HL2) while they performed three activities (i.e., read, inspect, search). We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved an up to 98.7% activity-recognition accuracy. On the HL2, we provided users with an AR feedback that is relevant to their current activity. We present the components of our system (GEAR) including a novel solution to enable the controlled sharing of collected data. We provide the scripts and anonymized datasets which can be used as teaching material in graduate courses or for reproducing our findings.

To enable people to interact more efficiently with virtual and physical services in their surroundings, it would be beneficial if information could more fluently be passed across digital and non-digital spaces. To this end, we propose to combine semantic technologies with Optical Character Recognition on an Augmented Reality (AR) interface to enable the semantic integration of (written) information located in our everyday environments with Internet of Things devices. We hence present SOCRAR, a system that is able to detect written information from a user’s physical environment while contextualizing this data through a semantic backend. The SOCRAR system enables in-band semantic translation on an AR interface, permits semantic filtering and selection of appropriate device interfaces, and provides cognitive offloading by enabling users to store information for later use. We demonstrate the feasibility of SOCRAR through the implementation of three concrete scenarios.

Nowadays, organizations include a large number of physical resources (e.g., meeting rooms, classrooms, scanners, and plotters) and virtual ones (e.g., multimedia files, applications, and databases) distributed among different offices, buildings, and storage devices. Typically, these resources have to be shared among colleagues, because it is impossible for each collaborator to own private instances of all the different resources present in the organization. In this way, resource sharing gives collaborators the opportunity of not just lending their resources to other collaborators but also benefitting from the usage of resources they do not own. However, finding shared resources in a huge organization, without a proper technological support, can be a challenge for a member of staff and obviously really hard or even impossible for an external person. To try to overcome this problem, several service discovery protocols have been developed, aiming to promote the use of network services and to reduce configuration tasks. Unfortunately, these protocols are mainly focused on finding ser- vices based just on their type or some minimal features. Such basic searches can be appropriate for services asking for services, but unlike applications, people expect to receive customized information concerning their identity, role, social relations, and even contextual variables (e.g., location). Moreover, people sharing resources (i.e., services) should have the certainty that their resources are not going to be in risk of abuse or bad management. Hence, in this work, we have taken the challenge of proposing a computational support capable of exploiting current available technologies, in order to provide people with a pervasive experience for resource sharing. Our proposal, the Resource Availability Management Service Architecture (RAMS), takes a semantic approach for finding the best available resources for a request, considering the interaction among people in- volved in the resource sharing process and the requester context. Such an interaction includes the evaluation of access rights, usage restrictions, owner relationships, resources, and even owners’ availability. The semantic approach of the RAMS Architecture brings multiple advantages when performing resources discovery, since non-evident relationships between entities can be computed, in order to offer a better and more accurate response for a person’s request. Such a semantic approach is being achieved by modeling the entities involved in the resource sharing process as a set of ontologies, which have been generically designed in order to be de- ployed in any size and type of organizations. Thus, this set of ontologies keeps information of such entities, which can correspond to technical characteristics and capabilities and some dynamic information about the environment. In the latter case, the dynamic information is retrieved from a human recognizer that determines a person’s presence and location by identifying his face and voice and from applications that allow people to change their current availability state or the one of their resources. The ubiquitous experience the RAMS Architecture offers is brought by such a human recognizer and by a resource locator, which in case of a physical resource discovery request, provides users with information about the closest resource available to him.