Apprentice at the End of the World
#immersive #theater #augmentedreality
I just wrapped an incredible production with students, staff and faculty of the UCLA School of Theater, Film and Television (TFT) and the Center for Research in Engineering, Media and Performance (REMAP). We created an immersive, ambulatory theater experience using augmented reality (AR) for social impact storytelling. I worked as a Creative Producer leading the production design (virtual and physical), the augmented reality design team, and the software development for the mobile application.
This production began as a course taught at TFT that discussed relevant work in AR mobile storytelling, interactive experiences and immersive theater.
The inspiration for the story was N.K. Jemisin's The Fifth Season, which I can honestly say is the best book I've read in a while. It deals with themes that are incredibly relevant to our current socio-political climate: colorism, oppression, the enslavement of people of color -- black people specifically -- and the volatility of dying earth (#climatechange).
After the students read and discussed the book, they pitched concepts for an AR experience based on The Fifth Season. After the quarter finished, they worked on a one-week writing workshop to develop the script and story references. In parallel, the staff pulled together supporting research and a minimum viable product for the mobile AR software.
We then began a three week intensive, the Future of Storytelling Summer Institute, on the stage of the Freud Playhouse at UCLA. We were an multidisciplinary group of directors, writers, designers, animators, performers, programmers and researchers.
The story world
(above) Exhibit design by Natalia Pavlova
Upon arrival, audience members onboarded together on the apron of the stage (ACT 1) and then separated into two groups (ACT 2). Afterword they reunited again in a climactic final scene in Castrima (ACT 3).
Overview of audience split in Act 2:
ACT 1: ONBOARDING
Prior to entering the experience, the audience convened outside in the courtyard of the Freud Playhouse. They were invited to read the exhibition materials about the experience and about our research questions. We did 10 open rehearsals with 8 participants in each run, as well as additional tech runs.
enter a post-apocalyptic world
Help participants use their devices for training, and help guide them through through the experience.
Participants are given a staff (with the device and AR application running), headphones and a subpack.
They explore their powers and learn how to "ground themselves" with their AR devices.
Fiducial anchors help the participants unlock information and media about the story world.
Contains lore from the world and, "fragments of information from Essun's fractured memories."
Floating crystal ruins with incredible power linked to Orogenes and ancient civilzations.
The 8 audience members are then separated into two groups, one with Syenite and another with Essun.
ACt 2: SPLIT PATHS
essun & Syenite
Two parts of the same whole: the same character at different points in her life. An orogene and a mother on a journey to find her last living child.
ESSUN's SEARCh FOR their DAUGHTER
syenite's militaristic training
Views From Different Perspectives
The two audience groups watch the center performances from either sides of the scrim.
Orogenic power overlaid on Essun, and other magical characters when they use their powers.
ACT 3: Ending in Castrima
The paths rejoin and the groups are together in an underground crystal city, Castrima, where they have to make a decision about the fate of the world: to fight the encroaching army or sacrifice some of their own to save the world.
A powerful Obelisk that is harnessed in order to save the world from the oncoming destruction.
AR views of the obelisk's unlocked power
software & systems
Initial app development:
Lead Developer - Peter Gusev;
Ilya a recent graduate from UCLA CS (left), myself (right)
Developed in unity
Integrating the models, textures, and placement of assets from student designers.
Mobile AR & projection
Both the overlaid media and the projection come from one virtual multiplayer world.
triggered & cued
AR capabilities were triggered via gesture and movement of the staffs, as well as cued via OSC.
Augmented Reality research
#experiential #design #research
Which elements of the story are most effective (in terms of audience engagement and/or enjoyment) when implemented 1) in the physical world; 2) in projection / media; 3) in AR delivered to the audience? How to manage/control attention across the three?
How do people interact with devices in an immersive atmosphere? Are we conditioned to pay more attention to our devices or to our surroundings? What does it take to get someone to look up from a device?
Which principles of interactive storytelling, experimental cinema, gaming, and/or immersive / audience-involved theater and performance are most useful when applied to AR, for our goals of impact / engagement?
How can the use of AR technology become part of the performance ritual rather than an intrusion? How can AR elements be imbued with significance from the outset, based on parallels to the uses of the devices in the real world?
How can AR be used to aid production itself? For example, could AR keep a map of lighting, i.e., magic sheet labels anchored to individual instruments, enabling designers to simply look up at fixtures and know channel/address/color/focus/etc.?
Texturing the stage
One of the biggest challenges of working in a blackbox theater for a computer vision project, is that often times there needs to be variations in color and texture to enable better tracking, and a space with uniform colored floors and walls do not provide stable tracking. As a result, the production design team devised a plan for treating the walls with projection and lighting, and the floor with paint as well as a symbol system for creating unique fiducial markers.
These symbols served as fiducial markers that helped anchor and align the virtual and physical environments, as well as a way to reveal secret information for audience members as they went through the experience. The left you can see the initial treatment with even lighting, on the right you can see the theatrical lighting with additional gobos, which added additional texture but did not hinder the tracking.
We received insightful feedback from audience members: they enjoyed the narrative and performances, felt disoriented when first using the technology, were unsure about when they should be looking at their phone or at their performance, and would have liked more time to explore the space to unlock the symbols.
How do we solve the 'inherent vice' of audience distraction with the device? What are some UX strategies for AR in immersive theater? How can AR facilitate communal experiences? How do we lean into the limitations of AR through incidental storytelling?
apprentice at the end of the world
july 9 - july 28, 2018 @ UCLA TFT
The 2018 Future of Storytelling Summer Institute was supported by funding from the Skoll Center for Social Impact Entertainment
at the UCLA School of Theater, Film and Television.