Summer Institute


Apprentice at the End of the World

#immersive #theater #augmentedreality

I just wrapped an incredible production with students, staff and faculty of the UCLA School of Theater, Film and Television (TFT) and the Center for Research in Engineering, Media and Performance (REMAP). We created an immersive, ambulatory theater experience using augmented reality (AR) for social impact storytelling. I worked as a Creative Producer leading the production design (virtual and physical), the augmented reality design team, and the software development for the mobile application.



This production began as a course taught at TFT that discussed relevant work in AR mobile storytelling, interactive experiences and immersive theater.

The inspiration for the story was N.K. Jemisin's The Fifth Season, which I can honestly say is the best book I've read in a while. It deals with themes that are incredibly relevant to our current socio-political climate: colorism, oppression, the enslavement of people of color -- black people specifically -- and the volatility of dying earth (#climatechange).

After the students read and discussed the book, they pitched concepts for an AR experience based on The Fifth Season. After the quarter finished, they worked on a one-week writing workshop to develop the script and story references. In parallel, the staff pulled together supporting research and a minimum viable product for the mobile AR software. 

We then began a three week intensive, the Future of Storytelling Summer Institute, on the stage of the Freud Playhouse at UCLA. We were an multidisciplinary group of directors, writers, designers, animators, performers, programmers and researchers.


The story world 

(above) Exhibit design by Natalia Pavlova

The Experience


audience flow

Upon arrival, audience members onboarded together on the apron of the stage (ACT 1) and then separated into two groups (ACT 2). Afterword they reunited again in a climactic final scene in Castrima (ACT 3).

Overview of audience split in Act 2: 



Prior to entering the experience, the audience convened outside in the courtyard of the Freud Playhouse. They were invited to read the exhibition materials about the experience and about our research questions.

We did 10 open rehearsals with 8 participants in each run, as well as additional tech runs. In all, we had over a 100 people see the production - woo!

enter a post-apocalyptic world



"Be greeted." by the ancient Stone Eater, Hoa.

Played by Rey Jarrell


Dark forces

Help participants use their devices for training, and help guide them through through the experience.

 Participants are given a staff (with the device and AR application running), headphones and a subpack. 

They explore their powers and learn how to "ground themselves" with their AR devices.



Fiducial anchors help the participants unlock information and media about the story world.



Contains lore from the world and, "fragments of information from Essun's fractured memories."



Floating crystal ruins with incredible power linked to Orogenes and ancient civilzations.


The 8 audience members are then separated into two groups, one with Syenite and another with Essun.

Participants with Syenite.



essun & Syenite

Two parts of the same whole: the same character at different points in her life. An orogene and a mother on a journey to find her last living child.


syenite's militaristic training 

Views From Different Perspectives

The two audience groups watch the center performances from either sides of the scrim.



Orogenic power overlaid on Essun, and other magical characters when they use their powers.

Screenshot view of Essun's torus icing the scene.

Screenshot view of Essun's torus icing the scene.

Ash falling from the sky of a post-apocalyptic world.

Ash falling from the sky of a post-apocalyptic world.

ACT 3: Ending in Castrima

The paths rejoin and the groups are together in an underground crystal city, Castrima, where they have to make a decision about the fate of the world: to fight the encroaching army or sacrifice some of their own to save the world. 


the onyx

A powerful Obelisk that is harnessed in order to save the world from the oncoming destruction.

AR views of the obelisk's unlocked power

Screenshot_20180728-164020_FifthAR Dev.jpg
Screenshot_20180728-164029_FifthAR Dev.jpg

software & systems

Initial app development:

Lead Developer - Peter Gusev;

Ilya a recent graduate from UCLA CS (left), myself (right)


Developed in unity

Integrating the models, textures, and placement of assets from student designers.


Mobile AR & projection

Both the overlaid media and the projection come from one virtual multiplayer world.


triggered & cued

AR capabilities were triggered via gesture and movement of the staffs, as well as cued via OSC.


Augmented Reality research

#experiential #design #research


  • Which elements of the story are most effective (in terms of audience engagement and/or enjoyment) when implemented 1) in the physical world; 2) in projection / media; 3) in AR delivered to the audience? How to manage/control attention across the three?

  • How do people interact with devices in an immersive atmosphere? Are we conditioned to pay more attention to our devices or to our surroundings? What does it take to get someone to look up from a device

  • Which principles of interactive storytelling, experimental cinema, gaming, and/or immersive / audience-involved theater and performance are most useful when applied to AR, for our goals of impact / engagement?

  • How can the use of AR technology become part of the performance ritual rather than an intrusion? How can AR elements be imbued with significance from the outset, based on parallels to the uses of the devices in the real world?

  • How can AR be used to aid production itself? For example, could AR keep a map of lighting, i.e., magic sheet labels anchored to individual instruments, enabling designers to simply look up at fixtures and know channel/address/color/focus/etc.?

Students of the 2018 FSS Institute

Students of the 2018 FSS Institute

Texturing the stage

One of the biggest challenges of working in a blackbox theater for a computer vision project, is that often times there needs to be variations in color and texture to enable better tracking, and a space with uniform colored floors and walls do not provide stable tracking. As a result, the production design team devised a plan for treating the walls with projection and lighting, and the floor with paint as well as a symbol system for creating unique fiducial markers.

fiducial anchors

These symbols served as fiducial markers that helped anchor and align the virtual and physical environments, as well as a way to reveal secret information for audience members as they went through the experience. The left you can see the initial treatment with even lighting, on the right you can see the theatrical lighting with additional gobos, which added additional texture but did not hinder the tracking.



Audience feedback

We received insightful feedback from audience members: they enjoyed the narrative and performances, felt disoriented when first using the technology, were unsure about when they should be looking at their phone or at their performance, and would have liked more time to explore the space to unlock the symbols.


new questions

How do we solve the 'inherent vice' of audience distraction with the device? What are some UX strategies for AR in immersive theater? How can AR facilitate communal experiences? How do we lean into the limitations of AR through incidental storytelling?


 * insert insightful commentary here #stillprocesing *


apprentice at the end of the world

july 9 - july 28, 2018 @ UCLA TFT

The 2018 Future of Storytelling Summer Institute was supported by funding from the Skoll Center for Social Impact Entertainment

at the UCLA School of Theater, Film and Television.

FSSI 2018 Draft 1 opt NOT FOR RELEASE.jpg