Problem statement
The current AI-revolution marks a new era in immersive storytelling within Extended Reality (XR). AI plays a crucial role in creating interactive stories, where the virtual world or storyline adapts to the user's choices. This transition from storytelling to "storyliving" allows users to actively participate in the story, enhancing their sense of agency and immersion. However, developing adaptive XR experiences presents creative, technical, and user experience (UX) challenges.
Although the gaming industry is already heavily invested in adaptive content, these techniques cannot be directly applied to Virtual Reality due to differences in UX requirements, such as presence and a sense of agency. It is important to investigate how these factors affect the user experience and which UX parameters are crucial for optimal engagement. Therefore, it is essential to map out new and existing storytelling techniques to effectively guide storytellers in these new forms of narration. A key part of this research is personalizing XR experiences based on personal input, such as using physiological data like heart rate. This can help create deeper immersion. While new generative AI models offer promising possibilities, hardware and software limitations remain a challenge for realizing fully adaptive XR experiences.
This research focuses on developing a framework for storyliving in XR, with attention to personalization and measuring effectiveness. This framework is valuable for producers in the entertainment, technology, and clinical sectors. The results will be compiled into a guide for the field at the end of the project.
Research questions
- Which new and existing storytelling techniques are essential for developing an adaptive XR experience to map out the parameters (e.g., narrative immersion) of storyliving?
- How can we adapt an XR experience based on personal input, such as questionnaires, behavioral observations, and psychophysiological measurements (e.g., galvanic skin response, heart rate, and eye movements)?
- What impact does adaptive XR content have on the user, and which UX parameters (e.g., sense of presence, sense of embodiment) have the most influence on their experience?
- What optimization techniques (e.g., culling, DOTS, LODS) are needed to enable a real-time adaptive XR experience on current hardware?
Research method
The desk research focuses on collecting best practices, mapping out measurement tools for user research, investigating parameters within storyliving, and comparing the pros and cons of tools for adaptive experiences.
We develop three proof-of-concepts, each aimed at answering multiple research questions. We investigate how to implement adaptive XR experiences easily and accessibly and how to involve AI in this process.
Through an experiment, one or more properties (e.g., story, environment, interactions with objects) of the XR experience are adjusted based on user input. During the experiment, we measure and evaluate the impact of the experience using both psychophysiological measures (attention measured via eye tracking, heart rate measured via hardware) and self-reporting (questionnaires with validated scales).
Expected results
- Overview of parameters and best practices for existing storytelling techniques, adaptive content in XR, and storyliving.
- Three proof-of-concepts, one of which is tested through an experiment.
- Guide for developing adaptive XR experiences.
- A technical workflow that allows a real-time adaptive XR world to function optimally.