Coexist: A mixed reality narrative installation
Crossing the physical-digital boundary through a holographic display and 3D interactions
Date
2022
Duration
6 weeks
Project Type
XR Narrative Installation
Keywords
Perspective-Calibrated, 3D Modeling, 3D Animation, Vive Tracker
Role
3D Modeling, Unity Prototyping, Laser Cutting, 3D Printing
Team
Aria Xiying Bao, Nix Liu Xin
Overview
Coexist is a mixed reality story expressed in the form of a screen-based installation. The viewer perceives a perspective-calibrated reality inside the physical screen and is able to interact with it through a flashlight. The light crosses the threshold between the two realities and shines into the virtual space, lighting up the growing tree.
This project is situated in a long lineage of research work on seeing 3D objects through screens. Acknowledging the paradigm shift brought by pioneering work in computer graphics, we referenced recent work exploring interactions for naked-eye stereoscopic displays and holographic displays, such as the ray-casting-based interactions by Egawa, Qiu, and Ijiri2, and interactions with a cubic 3D display by Lam and others.
Commercially, companies like Looking Glass Factory have been developing stereoscopic displays aimed for mass adoption4. While the hardware is iterating fast, research on 3D display interactions in the context of Extended Reality is limited at this moment and we see opportunities in exploring multi-modal mixed reality interactions combining both headsets and 3D displays.
Narrative
A disastrous event caused mass extinction and humans seem to have lost the ability to dream. Nella is a renowned botanist that often dreams about the plants in the pre-apocalyptic world. His life has been turned upside down like everybody else. Almost all of the botanical species Nella researches have gone extinct. Rather than studying live specimens, Nella now is a resident scientist at the United Nations responsible for leading the efforts of reviving extinct plant species. The dreams were never lost, but in fact aggregated to become a parallel universe. The two worlds are collapsing into one and the realities are
interpenetrating. One day, Nella picks up a flashlight during a power outage in his lab and shines through a specimen box…
Screen-Based XR Interactions
Custom 3D printed flashlight and lab glasses integrated with Vive spatial trackers
The flashlight and the lab glasses work together with the 3D scene in the screen to create digital effects from physical interactions. The digital scene is aware of the virtual position of the flashlight and the lab glasses so that the perspective adjusts in real time and the physical lighting can be mapped to a virtual light source.
Perspective-Calibrated Holographic Display Prototype
Scenes
The Botanical Lab
The Plant Existing in Two Realities
Interaction Flow
References
Kittler, Friedrich A.. “Computer Graphics: A Semi-Technical Introduction.” Grey Room (2001).
Raku Egawa, Yucheng Qiu, and Takashi Ijiri. 2020. Ray-Casting-based 3D pointing and Dragging Interface for
Naked-Eye Stereoscopic Displays. In 26th ACM Symposium on Virtual Reality Software and Technology (VRST '20).
Association for Computing Machinery, New York, NY, USA, Article 69, 1–3.
https://doi.org/10.1145/3385956.3422102
Billy Lam, Ian Stavness, Ryan Barr, and Sidney Fels. 2009. Interacting with a personal cubic 3D display. In
Proceedings of the 17th ACM international conference on Multimedia (MM '09). Association for Computing
Machinery, New York, NY, USA, 959–960. https://doi.org/10.1145/1631272.1631467
Alex N. Hornstein, Evan Moore, and Kai-han Chang. 2019. A Match Made in Heaven: Streaming Real-time Imagery
from a Lightfield Camera to a Lightfield Display. In The Adjunct Publication of the 32nd Annual ACM Symposium on
User Interface Software and Technology (UIST '19). Association for Computing Machinery, New York, NY, USA,
140–141. https://doi.org/10.1145/3332167.3356899