Seasons

The project explores embodied interaction in virtual environments and how physical and digital content can work together in an experiential system to enhance collaborative learning. I built a physical globe as an input device. Participants can touch specific locations on the globe, triggering different virtual environments that feature local geographical and climatic characteristics. They can then interact with natural elements in these virtual environments using their bodies. This project includes two cycles of development and testing. Below are the documentations for this project:

For this cycle, I wanted to create interactive virtual environments that contribute to embodied learning. I chose Seasons as the theme of this project. Participants can interact with natural elements specific to each season using their bodies. For example, spring with flowers, summer with rain, autumn with falling leaves, and winter with snow.

Using a webcam as the input device, I utilized various TouchDesigner features:
‍NVbackground: This actor removes the background.
‍Optical flow: This actor tracks body movements.
‍ParticleGup: This feature allows the creation of different interactive particles.

Since I was new to TouchDesigner, I spent a considerable amount of time figuring out how to transition from one scene to another in a sequential manner, a task that is relatively simple in Isadora. In this cycle, I managed to create only the autumn and winter scenes. I am still learning about other interactive visual effects using different body languages and how to trigger sounds through movements in TouchDesigner. Below is the TouchDesigner patch:

I was excited to install the project in the motion lab. To avoid shadows, Oded suggested projecting the image onto the back of the screen, with people standing in front of the screen to interact with the virtual environments. It was magical to see both digital figures and human shadows reflected on the curtain from people standing before and behind it.

Participants enjoyed interacting with the snow and leaves. However, due to the limitations of the webcam, they had to stand in a specific area to stay within the screen's frame. Next time, I plan to try using a Kinect sensor to see if it solves this issue. Additionally, I learned that lighting is crucial as it affects the quality of the real-time input video.

Shadrick provided some valuable suggestions for further developing this learning system. First, I can add fun filters to the digital figures, such as giving them hats in winter. Second, I can create environments based on different regions, as different countries have varying climatic characteristics. In the next cycle, I will also look to combine media-enhanced objects and environments to enrich the interactive experience.

If the video doesn’t load, you can watch them on Bilibili instead.
Season_Cycle1: https://www.bilibili.com/video/BV1aGx2erEDs/

In the second cycle, I constructed a spherical interface using Makey Makey, copper tape, and conductive paint. Participants could experience virtual environments of specific locations on the globe by touching marked signs. I researched countries and places that were in different seasons when I make this project (late April). I selected Tokyo (Japan) for spring, the Amazon rainforest (Brazil) for summer, Mount Macedon (Australia) for autumn, and Yellowknife (Canada) for winter, marking these locations on the globe.

This time, I used a Kinect sensor, which worked much better than the webcam. In the Spring Tokyo scene, the Kinect tracks participants' heads, causing flowers to fall above them. In the Summer Amazon rainforest scene, rain falls on people and bounces off. I plan to explore more interactive visual effects in the future. Additionally, I incorporated audio into each scene, enhancing the immersive experience.

If the videos don’t load, you can watch them on Bilibili instead.
Season_Cycle2.1: https://www.bilibili.com/video/BV1Y3x2eGE44/
Season_Cycle2.2: https://www.bilibili.com/video/BV1Yux2eCE1U/